Changes between Version 8 and Version 9 of Projects/OpenIFS-IO

Timestamp:

24/09/13 12:39:47 (8 years ago)

Author:

grenville

Comment:

—

Projects/OpenIFS-IO

@@ -41 +41 @@
 
 
-=== Check out the OIFS code ===
+=== Setting the HECToR environment ===
 
-You will need to set up the correct environment to build and run OIFS jobs. Since HECToR has different hardware on its compute and service nodes, there is an extra consideration when using some OIFS-related utilities on the service nodes (more later). Let's assume that you are interested in building and running the model only for now. At the HECToR command line, type
+You will need to set up the correct environment to build and run OIFS jobs. Since HECToR has different hardware on its compute and service nodes, there is an extra consideration when using some OIFS-related utilities on the service nodes (more later). We have written a script which sets up the environment, however, it is worth taking a look at its main component. At the HECToR command line, type
 
 {{{
@@ -74 +74 @@
 If your normal module set up is with the gnu or pgi programming environments, then an attempt to load openifs_cce/0.0.2 will fail, and it will also fail unless you have the Cray cce1/8.1.8 compiler loaded. Use '''module swap''' to ensure an appropriate starting module configuration.
 
+=== Check out the OIFS code ===
 
 On HECToR create a suitably named directory in your /home space. We recommend building in /home which handles small files more efficiently than does /work for a quicker build. Now check out the OIFS code tree with '''fcm co'''. Here is an example
@@ -93 +94 @@
 }}}
 
-this runs the compile and link phases of the build, using 4 processors (-j 4), with verbose output (-vv) based on the configuration file in cfg/oifs.cfg (the argument of -f option) (see the FCM documentation for more options for the fcm make command). This may take some time.
+this runs the compile and link phases of the build, using 4 processors (-j 4), with verbose output (-vv) based on the configuration file in cfg/oifs.cfg (the argument of -f option) (see the FCM documentation for more options for the fcm make command). This command may take some time to complete.
 
 The executable, currently called master.exe, will be in make/opt/oifs/bin.
@@ -99 +100 @@
 === Getting data to run the model ===
 
-The source for data is Glenn Carver at ECMWF. Glenn can provide start data for configurations at several resolutions and for various times. We have several jobs available for model testing at t159, t511 and t1279 resolution. Each job comes with data required to start the model, a namelist with a set of model parameters, a PBS script set up for a "standard" run and an example trace output file. We will give a description of the PBS script used to submit jobs to the computer nodes.
+Now you have an executable, you will need some data. The source for data is Glenn Carver at ECMWF; Glenn can provide start data for configurations at several resolutions and for various times. We have several jobs available for model testing at t159, t511 and t1279 resolution. Each job comes with data required to start the model, a namelist defining a set of model parameters, a PBS script set up for a "standard" run and an example trace output file. We will give a description of the important parts of the PBS script used to submit jobs to the compute nodes.
 
+The file has the usual PBS directives for reserving HPC resource, in this case we have requested 1024 processors with fully populated nodes, for a time limit of 25 minutes. You should edit this to specify your account and email details and change the HPC resource request as appropriate.
 {{{
 #!/bin/bash --login
@@ -119 +121 @@
 #PBS -M g.m.s.lister@reading.ac.uk
 #PBS -m ae
+}}}
+
+The remainder of the PBS script refers more specifically to your job. oifs.setup is the script which loads the appropriate oifs modules. Please copy one from ...[we should put a template in the repository].
 
 
+{{{
 # Set environment
 source $HOME/oifs.setup
@@ -129 +135 @@
 export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR)
 
-# Chnage to the directory the job was submitted from
+# Change to the directory the job was submitted from
 cd $PBS_O_WORKDIR
+}}}
+
+Currently, you will need to set OIFS_HOME to your oifs code tree
+{{{
 
 #  Root directory of OpenIFS
-#export OIFS_HOME=$HOME/oifs38r1v01
-#  make sure correct branch is active
-#export OIFS_HOME=$HOME/git/openifs
 export OIFS_HOME=$HOME/oifs_extract/extract/oifs
 
 #  Set the GRIB_SAMPLES_PATH to your installation
-#  set DEFINITION_PATH so grib_api can find boot.def 
-#  set OIFS_GRIB_API_DIR to point to local installation
 export GRIB_SAMPLES_PATH=$GRIB_API_PATH/share/grib_api/ifs_samples/grib1_mlgrib2
-export GRIB_DEFINITION_PATH=$GRIB_API_PATH/share/grib_api/definitions
 
 #  use assign statement to read f77 unformatted files as big-endian
@@ -148 +152 @@
 export FILENV=$PBS_O_WORKDIR/.myassign
 assign -N swap_endian g:su
+}}}
 
+The following section allows you to specify the compute decomposition - this example will use 128 MPI tasks (NPROC), each with 8 OMP threads (NTHREADS). Note NPROC x NTHREADS = 1024 which was the number specified for mppwidth in the header section of the file. EXPID refers to the ECMWF supplied experiment id. RUN is simply a means of distinguishing output from different runs (the names of output directories will be distinguished by RUN). TSTEP is the model time step in seconds (the best value has been determined by ECMWF in general). You probably won't change MASTER or NAMELIST. 
+
+The script ifs_run.sh sets up the aprun command based on its arguments. The model resolution is specified through -r (511 in this case) and the length of the model run (in model days) is specified through the -f option 
+
+
+{{{
 # Run model
 EXPID=fw13