Changes between Version 8 and Version 9 of Projects/OpenIFS-IO


Ignore:
Timestamp:
24/09/13 12:39:47 (6 years ago)
Author:
grenville
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Projects/OpenIFS-IO

    v8 v9  
    4141 
    4242 
    43 === Check out the OIFS code === 
     43=== Setting the HECToR environment === 
    4444 
    45 You will need to set up the correct environment to build and run OIFS jobs. Since HECToR has different hardware on its compute and service nodes, there is an extra consideration when using some OIFS-related utilities on the service nodes (more later). Let's assume that you are interested in building and running the model only for now. At the HECToR command line, type 
     45You will need to set up the correct environment to build and run OIFS jobs. Since HECToR has different hardware on its compute and service nodes, there is an extra consideration when using some OIFS-related utilities on the service nodes (more later). We have written a script which sets up the environment, however, it is worth taking a look at its main component. At the HECToR command line, type 
    4646 
    4747{{{ 
     
    7474If your normal module set up is with the gnu or pgi programming environments, then an attempt to load openifs_cce/0.0.2 will fail, and it will also fail unless you have the Cray cce1/8.1.8 compiler loaded. Use '''module swap''' to ensure an appropriate starting module configuration. 
    7575 
     76=== Check out the OIFS code === 
    7677 
    7778On HECToR create a suitably named directory in your /home space. We recommend building in /home which handles small files more efficiently than does /work for a quicker build. Now check out the OIFS code tree with '''fcm co'''. Here is an example 
     
    9394}}} 
    9495 
    95 this runs the compile and link phases of the build, using 4 processors (-j 4), with verbose output (-vv) based on the configuration file in cfg/oifs.cfg (the argument of -f option) (see the FCM documentation for more options for the fcm make command). This may take some time. 
     96this runs the compile and link phases of the build, using 4 processors (-j 4), with verbose output (-vv) based on the configuration file in cfg/oifs.cfg (the argument of -f option) (see the FCM documentation for more options for the fcm make command). This command may take some time to complete. 
    9697 
    9798The executable, currently called master.exe, will be in make/opt/oifs/bin. 
     
    99100=== Getting data to run the model === 
    100101 
    101 The source for data is Glenn Carver at ECMWF. Glenn can provide start data for configurations at several resolutions and for various times. We have several jobs available for model testing at t159, t511 and t1279 resolution. Each job comes with data required to start the model, a namelist with a set of model parameters, a PBS script set up for a "standard" run and an example trace output file. We will give a description of the PBS script used to submit jobs to the computer nodes. 
     102Now you have an executable, you will need some data. The source for data is Glenn Carver at ECMWF; Glenn can provide start data for configurations at several resolutions and for various times. We have several jobs available for model testing at t159, t511 and t1279 resolution. Each job comes with data required to start the model, a namelist defining a set of model parameters, a PBS script set up for a "standard" run and an example trace output file. We will give a description of the important parts of the PBS script used to submit jobs to the compute nodes. 
    102103 
     104The file has the usual PBS directives for reserving HPC resource, in this case we have requested 1024 processors with fully populated nodes, for a time limit of 25 minutes. You should edit this to specify your account and email details and change the HPC resource request as appropriate. 
    103105{{{ 
    104106#!/bin/bash --login 
     
    119121#PBS -M g.m.s.lister@reading.ac.uk 
    120122#PBS -m ae 
     123}}} 
     124 
     125The remainder of the PBS script refers more specifically to your job. oifs.setup is the script which loads the appropriate oifs modules. Please copy one from ...[we should put a template in the repository]. 
    121126 
    122127 
     128{{{ 
    123129# Set environment 
    124130source $HOME/oifs.setup 
     
    129135export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR) 
    130136 
    131 # Chnage to the directory the job was submitted from 
     137# Change to the directory the job was submitted from 
    132138cd $PBS_O_WORKDIR 
     139}}} 
     140 
     141Currently, you will need to set OIFS_HOME to your oifs code tree 
     142{{{ 
    133143 
    134144#  Root directory of OpenIFS 
    135 #export OIFS_HOME=$HOME/oifs38r1v01 
    136 #  make sure correct branch is active 
    137 #export OIFS_HOME=$HOME/git/openifs 
    138145export OIFS_HOME=$HOME/oifs_extract/extract/oifs 
    139146 
    140147#  Set the GRIB_SAMPLES_PATH to your installation 
    141 #  set DEFINITION_PATH so grib_api can find boot.def  
    142 #  set OIFS_GRIB_API_DIR to point to local installation 
    143148export GRIB_SAMPLES_PATH=$GRIB_API_PATH/share/grib_api/ifs_samples/grib1_mlgrib2 
    144 export GRIB_DEFINITION_PATH=$GRIB_API_PATH/share/grib_api/definitions 
    145149 
    146150#  use assign statement to read f77 unformatted files as big-endian 
     
    148152export FILENV=$PBS_O_WORKDIR/.myassign 
    149153assign -N swap_endian g:su 
     154}}} 
    150155 
     156The following section allows you to specify the compute decomposition - this example will use 128 MPI tasks (NPROC), each with 8 OMP threads (NTHREADS). Note NPROC x NTHREADS = 1024 which was the number specified for mppwidth in the header section of the file. EXPID refers to the ECMWF supplied experiment id. RUN is simply a means of distinguishing output from different runs (the names of output directories will be distinguished by RUN). TSTEP is the model time step in seconds (the best value has been determined by ECMWF in general). You probably won't change MASTER or NAMELIST.  
     157 
     158The script ifs_run.sh sets up the aprun command based on its arguments. The model resolution is specified through -r (511 in this case) and the length of the model run (in model days) is specified through the -f option  
     159 
     160 
     161{{{ 
    151162# Run model 
    152163EXPID=fw13