wiki:Projects/INSPECT/NEMOTAM

NEMOTAM test run

12 August 2013

We installed and ran the example ORCA2 NEMO-TAM experiment provided with the NEMO 3.4.1 source code on one of the local machines (um1, identical to um2). We followed the instructions listed on the main NEMO website: http://www.nemo-ocean.eu/Using-NEMO/User-Guides/Basics/Tangent-and-Adjoint-quick-start-guide

Information and instructions for running locally are given below:

Input data

The input files are located here: /net/jasmin/users/annette/nemo/ORCA2_LIM_nemo_v3.4

This is the ORCA2_LIM v3.4 data set mentioned here:

http://www.nemo-ocean.eu/Using-NEMO/Configurations/ORCA2_LIM-and-AGRIF-demonstrator

Timing runs

The times taken to run 75 timesteps (5 model days), based on the 'time' command around the model executable. This includes writing of restart files but not diagnostic history files.

NEMO

1 proc  -> 3m17.43s
4 proc  -> 1m6.12s
8 proc  -> 0m55.86s
16 proc -> 1m29.75s

TAM

1 proc  -> 1m29.24s
4 proc  -> 0m34.25s
8 proc  -> ERROR
16 proc -> 0m43.27s

Instructions for running locally

Step 1) Set up environment for um[12]

You should be on the local machine on which you wish to run the experiment.

a) Add the following lines to your .kshrc file:

if [ `uname -n` = 'um1' -o `uname -n` = 'um2' ]
then 
  ulimit -c unlimited
  ulimit -s unlimited
  . /net/jasmin/users/simon/intel/composerxe\-2011.4.191/bin/compilervars.sh intel64
  export PATH=/home/jeff/linux_x86_64/mpich2-install/bin:$PATH
  export LD_LIBRARY_PATH=/panfs/jasmin/users/annette/netcdf/intel/lib:$LD_LIBRARY_PATH   
  export PATH=/net/jasmin/users/annette/fcm/bin:$PATH
  export PATH=/net/jasmin/users/annette/nemo/rebuild:$PATH
fi

This ensures the supporting software is available.

Step 2) Extract code to um servers

You need to be on puma for this step.

a) Create a directory to extract NEMO code to eg:

mkdir ~/nemotam/v3.4.1_test
cd ~/nemotam/v3.4.1_test

b) Copy the following fcm configuration file:

cp ~annette/nemotam/v3.4.1_test/fcm-make.cfg . 

This contains all the information about where to get the source code from and how to build it. For more information see: http://cms.ncas.ac.uk/chrome/site/FCM/user_guide/make.html

c) Edit the mirror.target line in the file. This should use your own username and point to an experiment directory that can be seen on the local um machines.

mirror.target = annette@remote:/path/to/data/space/v3.4.1_test

d) Run extract (ignore warnings about SYMLINKS). This should be done in the directory where the fcm-make.cfg file sits.

fcm make 

This extracts the v3.4.1 branch and copies the source code over to your local data-space.

Step 3) Build ORCA2 NEMO and NEMOTAM on um servers

Log in to the um servers.

a) Go to the experiment directory created by the extract step:

cd /net/jasmin/users/annette/nemotam/v3.4.1_test

You should see another fcm-make.cfg file. This one just includes the build information, and you should not change anything in here.

b) Build the model executables:

fcm make -j 8 -v

This uses 8 parallel threads.

Step 4) Run direct NEMO model

Assuming the build completed successfully you should have two executables in the build/bin directory: nemo.exe and nemo_tam.exe.

a) Create an experiment directory to perform the run.

cd /path/to/data
mkdir v3.4.1_test_run
cd v3.4.1_test_run

In this example we assume this is at the same level as the v3.4.1_test directory created for the code, for example:

$ ls ../
OLD/  ORCA2_TAM_EXAMPLE/  v3.4.1_test/  v3.4.1_test_run/

b) Copy over example run scripts and control files:

cp /net/jasmin/users/annette/nemotam/ORCA2_TAM_EXAMPLE/* . 

c) Edit the run_opa.ksh file to point to the executables you have just built, eg:

EXEDIR=/net/jasmin/users/annette/nemotam/v3.4.1_test/build/bin

This sets the model to run for 75 timesteps (5 days) on 4 processors.

d) Run the direct NEMO model first:

./run_opa.ksh 

It should take just over a minute to complete.

NEMO produces netcdf output in the form of restart files and diagnostic files (labelled grid_[TUVW]). In this case it also writes *tam_trajectory* files to feed into TAM. Note that the diagnostic files will be empty as the run is too short.

Output is written per processor, so to concatenate the files use the rebuild_nemo command. For example to generate a global restart file, run:

rebuild_nemo ORCA2_00000075_restart 4

General model information is written to ocean.output.DIRECT.

Step 5) Run NEMO-TAM model

NEMO TAM should be run in the same directory that the direct model was run, or at least should have the same input files, namelist and the tam trajectory files produced by the direct model.

a) Edit the run_tam.ksh file to point to the executable directory, eg:

EXEDIR=/net/jasmin/users/annette/nemotam/v3.4.1_test/build/bin

The number of processors should remain the same as for the direct model.

b) Now run TAM:

./run_tam.ksh . 

TAM produces tl_trajectory files. Model output information is written to a file ocean.output.TAM.

Last modified 4 years ago Last modified on 09/04/15 12:37:47