Changes between Version 8 and Version 9 of Projects/OpenIFS-IO
- Timestamp:
- 24/09/13 12:39:47 (7 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Projects/OpenIFS-IO
v8 v9 41 41 42 42 43 === Check out the OIFS code===43 === Setting the HECToR environment === 44 44 45 You will need to set up the correct environment to build and run OIFS jobs. Since HECToR has different hardware on its compute and service nodes, there is an extra consideration when using some OIFS-related utilities on the service nodes (more later). Let's assume that you are interested in building and running the model only for now. At the HECToR command line, type45 You will need to set up the correct environment to build and run OIFS jobs. Since HECToR has different hardware on its compute and service nodes, there is an extra consideration when using some OIFS-related utilities on the service nodes (more later). We have written a script which sets up the environment, however, it is worth taking a look at its main component. At the HECToR command line, type 46 46 47 47 {{{ … … 74 74 If your normal module set up is with the gnu or pgi programming environments, then an attempt to load openifs_cce/0.0.2 will fail, and it will also fail unless you have the Cray cce1/8.1.8 compiler loaded. Use '''module swap''' to ensure an appropriate starting module configuration. 75 75 76 === Check out the OIFS code === 76 77 77 78 On HECToR create a suitably named directory in your /home space. We recommend building in /home which handles small files more efficiently than does /work for a quicker build. Now check out the OIFS code tree with '''fcm co'''. Here is an example … … 93 94 }}} 94 95 95 this runs the compile and link phases of the build, using 4 processors (-j 4), with verbose output (-vv) based on the configuration file in cfg/oifs.cfg (the argument of -f option) (see the FCM documentation for more options for the fcm make command). This may take some time.96 this runs the compile and link phases of the build, using 4 processors (-j 4), with verbose output (-vv) based on the configuration file in cfg/oifs.cfg (the argument of -f option) (see the FCM documentation for more options for the fcm make command). This command may take some time to complete. 96 97 97 98 The executable, currently called master.exe, will be in make/opt/oifs/bin. … … 99 100 === Getting data to run the model === 100 101 101 The source for data is Glenn Carver at ECMWF. Glenn can provide start data for configurations at several resolutions and for various times. We have several jobs available for model testing at t159, t511 and t1279 resolution. Each job comes with data required to start the model, a namelist with a set of model parameters, a PBS script set up for a "standard" run and an example trace output file. We will give a description of the PBS script used to submit jobs to the computernodes.102 Now you have an executable, you will need some data. The source for data is Glenn Carver at ECMWF; Glenn can provide start data for configurations at several resolutions and for various times. We have several jobs available for model testing at t159, t511 and t1279 resolution. Each job comes with data required to start the model, a namelist defining a set of model parameters, a PBS script set up for a "standard" run and an example trace output file. We will give a description of the important parts of the PBS script used to submit jobs to the compute nodes. 102 103 104 The file has the usual PBS directives for reserving HPC resource, in this case we have requested 1024 processors with fully populated nodes, for a time limit of 25 minutes. You should edit this to specify your account and email details and change the HPC resource request as appropriate. 103 105 {{{ 104 106 #!/bin/bash --login … … 119 121 #PBS -M g.m.s.lister@reading.ac.uk 120 122 #PBS -m ae 123 }}} 124 125 The remainder of the PBS script refers more specifically to your job. oifs.setup is the script which loads the appropriate oifs modules. Please copy one from ...[we should put a template in the repository]. 121 126 122 127 128 {{{ 123 129 # Set environment 124 130 source $HOME/oifs.setup … … 129 135 export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR) 130 136 131 # Ch nage to the directory the job was submitted from137 # Change to the directory the job was submitted from 132 138 cd $PBS_O_WORKDIR 139 }}} 140 141 Currently, you will need to set OIFS_HOME to your oifs code tree 142 {{{ 133 143 134 144 # Root directory of OpenIFS 135 #export OIFS_HOME=$HOME/oifs38r1v01136 # make sure correct branch is active137 #export OIFS_HOME=$HOME/git/openifs138 145 export OIFS_HOME=$HOME/oifs_extract/extract/oifs 139 146 140 147 # Set the GRIB_SAMPLES_PATH to your installation 141 # set DEFINITION_PATH so grib_api can find boot.def142 # set OIFS_GRIB_API_DIR to point to local installation143 148 export GRIB_SAMPLES_PATH=$GRIB_API_PATH/share/grib_api/ifs_samples/grib1_mlgrib2 144 export GRIB_DEFINITION_PATH=$GRIB_API_PATH/share/grib_api/definitions145 149 146 150 # use assign statement to read f77 unformatted files as big-endian … … 148 152 export FILENV=$PBS_O_WORKDIR/.myassign 149 153 assign -N swap_endian g:su 154 }}} 150 155 156 The following section allows you to specify the compute decomposition - this example will use 128 MPI tasks (NPROC), each with 8 OMP threads (NTHREADS). Note NPROC x NTHREADS = 1024 which was the number specified for mppwidth in the header section of the file. EXPID refers to the ECMWF supplied experiment id. RUN is simply a means of distinguishing output from different runs (the names of output directories will be distinguished by RUN). TSTEP is the model time step in seconds (the best value has been determined by ECMWF in general). You probably won't change MASTER or NAMELIST. 157 158 The script ifs_run.sh sets up the aprun command based on its arguments. The model resolution is specified through -r (511 in this case) and the length of the model run (in model days) is specified through the -f option 159 160 161 {{{ 151 162 # Run model 152 163 EXPID=fw13