Changes between Version 25 and Version 26 of Docs/Polaris


Ignore:
Timestamp:
03/07/15 16:14:00 (4 years ago)
Author:
ros
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Docs/Polaris

    v25 v26  
    1  
     1{{{#!comment 
    22{{{#!box 
    33This page is under construction and details presented here are likely to change 
    44}}} 
    5  
     5}}} 
    66[[CmsSiteToc]] 
    77 
    88= Setting up UM 6.6.3 to run on Polaris = 
    99 
    10 == 1 Central Installation Directory Structure == 
    11 The central UM directory is /home/polaris_lds1/earhum. FCM (code management system), gcom (UM communications software), and UM vn 6.6.3 are held in this directory. Other versions of the UM and related software will also be installed here.  
    12  
    13 The central installation of UM vn 6.6.3 is in /home/polaris_lds1/earhum/hg6.6.3.  
     10== 1. Central Installation Directory Structure == 
     11The central UM directory is `/home/polaris_lds1/earhum`. FCM (code management system), gcom (UM communications software), and UM vn 6.6.3 are held in this directory. Other versions of the UM and related software will also be installed here.  
     12 
     13The central installation of UM vn 6.6.3 is in `/home/polaris_lds1/earhum/hg6.6.3` 
    1414||'''Directory'''||'''Description of Contents'''|| 
    1515||HG2ES_ancils||Ancillary files for the HadGEM2-ES model|| 
     
    2121||sgi||small executables and UM utilities, and installation information|| 
    2222 
    23 == 2 Environment Variables and Your Files == 
     23== 2. Environment Variables and Your Files == 
    2424Standard UM environment variables are set thus 
    2525||'''Environment Variable'''||'''Value'''|| 
     
    3333 
    3434 
    35 so dumps, diagnostics, lbcs, history files, and intermediate output will by default be sent to /nobackup/$USER/$RUNID. The final leave file for a run will be saved in /home/polaris_lds1/$USER/output. 
     35so dumps, diagnostics, lbcs, history files, and intermediate output will by default be sent to `/nobackup/$USER/$RUNID`. The final leave file for a run will be saved in `/home/polaris_lds1/$USER/output`. 
    3636 
    3737 
    3838'''Start files. ''' 
    39 Example start files for the four HadGEM2 jobs are provided. We have mimicked the file structure as it exists on HECToR, so start files for HadGEM2-ES are in $UMDIR/hg6.6.3/dumps, all other models are provided with example start files along with their ancillary files. 
    40  
    41 == 3 Your Setup on Polaris == 
    42 Copy the UM Setup section from /home/polaris_lds1/ldsgl/.bashrc to your .bashrc (note, if you wish to use utilities for a version of the UM other than 6.6.3, you will need to source the appropriate .umsetvars file) 
    43  
    44 == 4 Model Build Information and UMUI Settings == 
     39Example start files for the four HadGEM2 jobs are provided. We have mimicked the file structure as it exists on HECToR, so start files for HadGEM2-ES are in `$UMDIR/hg6.6.3/dumps`, all other models are provided with example start files along with their ancillary files. 
     40 
     41== 3. Your Setup on Polaris == 
     42Copy the UM Setup section from `/home/polaris_lds1/ldsgl/.bashrc` to your `.bashrc` (note, if you wish to use utilities for a version of the UM other than 6.6.3, you will need to source the appropriate `.umsetvars` file) 
     43 
     44== 4. Model Build Information and UMUI Settings == 
    4545The model build information is kept in configuration files which are read by the FCM build system. The appropriate compiler flags are used to build the model along with information about which libraries to link. We have used the following flags for the Polaris intel compiler for both model and communications builds 
    4646 
     
    7777Choose the option ''''qsub' for SGE(SGI)''' for the submission method. The '''Job memory limit''' is not relevant (all jobs currently request the default memory/core of 4GB). 
    7878 
    79 '''Gotchas for HadGEM2-AMIP''' 
     79'''Gotchas for HadGEM2-AMIP:''' \\ 
    8080At two places in the UMUI, files need to be specified by full paths: navigate to ''model selection -> atmosphere -> ancillary and input data files -> climatologies and potential climatologies -> natural climate forcing'' and specify /home/polaris_lds1/earhum/hg6.6.3/HG2AMIP_ancils as the path for the '''solar forcing'''  and '''volcanic forcing''' files. 
    8181 
    8282We have undertaken limited testing of Climate Meaning to find that inclusion of STASH item 262 (section 0) BOUNDARY LAYER CONVECTION FLAG causes a checksum failure. Switch off this stash item when running with Climate Meaning. 
    8383 
    84 == 5 Example UMUI Jobs == 
     84== 5. Example UMUI Jobs == 
    8585The following jobs are currently under user grenville in the UMUI 
    8686||Job Id||  Model  || 
     
    9191 
    9292 
    93 == 6 Performance and Scaling == #Performance 
     93== 6. Performance and Scaling == #Performance 
    9494 
    9595 
    9696The following figure summarizes results from test running the four HadGEM2 jobs with increasing processor count. The jobs were run for 3 model days, with the exception of HadGEM2-CC was run for 1 model day. The results for total wall clock time for HadGEM2-ES running on HECToR and CURIE (a PRACE machine (Bull)) are also shown. Data points in red refer to the compute time for the Polaris jobs. All models display a lack of scaling when run on 96 processors. 
    9797 
    98 [[Image(hg2.png)]] 
     98[[Image(hg2.png, 800px, center)]] 
     99 
     100---- 
    99101 
    100102= Setting up UM 7.3 to run on Polaris = 
    101103 
    102 == 1 Central Installation Directory Structure == 
    103 The vn 7.3 installation is under /home/polaris_lds1/earhum/vn7.3/. The subdirectory HGPKG2 holds all the data files necessary to run the standard HadGEM3 job xidef (the same job on HECToR is xeozb). 
     104== 1. Central Installation Directory Structure == 
     105The vn 7.3 installation is under `/home/polaris_lds1/earhum/vn7.3`. The subdirectory HGPKG2 holds all the data files necessary to run the standard HadGEM3 job xidef (the same job on HECToR is xeozb). 
    104106The various other subdirectorties hold control data for the model, and data to run xideh (a copy of xfvgc). 
    105107 
    106 == 2 Your Setup on Polaris == 
    107 Copy the UM Setup section from /home/polaris_lds1/ldsgl/.bashrc to your .bashrc (note, if you wish to use utilities for a version of the UM other than 6.6.3, you will need to source the appropriate .umsetvars file) 
    108  
    109 == 3 Model Build Information and UMUI Settings == 
     108== 2. Your Setup on Polaris == 
     109Copy the UM Setup section from `/home/polaris_lds1/ldsgl/.bashrc` to your `.bashrc` (note, if you wish to use utilities for a version of the UM other than 6.6.3, you will need to source the appropriate .umsetvars file) 
     110 
     111== 3. Model Build Information and UMUI Settings == 
    110112The model build information is kept in configuration files which are read by the FCM build system. The appropriate compiler flags are used to build the model along with information about which libraries to link. We have used the following flags for the Polaris intel compiler for both model and communications builds 
    111113 
    112114{{{-i8 -r8 -fp-model precise -O1}}} 
    113115 
    114 UM vn 7.3 requires the inclusion of override files to correctly set compiler and linker options. Navigate to ''model selection-> compile and modifications-> UM user override files'' and include /home/grenville/umui_jobs/overrides/polaris_7.3_machine and /home/grenville/umui_jobs/overrides/polaris_7.3_file in the tables User machine overrides and User file overrides respectively. 
     116UM vn 7.3 requires the inclusion of override files to correctly set compiler and linker options. Navigate to ''model selection-> compile and modifications-> UM user override files'' and include `/home/grenville/umui_jobs/overrides/polaris_7.3_machine` and `/home/grenville/umui_jobs/overrides/polaris_7.3_file` in the tables '''User machine overrides''' and '''User file overrides''' respectively. 
    115117 
    116118 
     
    131133 
    132134Navigate to ''model selection -> sub-model independent -> FCM configuration ->  FCM configuration variables''. 
    133 Experience will help determine the most convenient places where the extracted model should reside. Explicitly set '''UM_ROUTDIR''' to be your user directory on /nobackup, however, note that files left in this directory may be marked for deletion in accordance with Polaris data policy. 
     135Experience will help determine the most convenient places where the extracted model should reside. Explicitly set '''UM_ROUTDIR''' to be your user directory on `/nobackup`, however, note that files left in this directory may be marked for deletion in accordance with Polaris data policy. 
    134136 
    135137Navigate to ''model selection -> user information and target machine -> general details'' 
     
    137139 
    138140Navigate to ''model selection -> user information and target machine -> target machine'' 
    139 Chose 'other' for the '''Compile, Link and Run the job''' option, and set the '''Other machine name''' to polaris.leeds.ac.uk 
     141Chose 'other' for the '''Compile, Link and Run the job''' option, and set the '''Other machine name''' to `polaris.leeds.ac.uk` 
    140142 
    141143Navigate to ''model selection -> input/output control and resources -> job submission, resources and resubmission pattern'' 
    142144Choose the option ''''qsub' for SGE(SGI)''' for the submission method. The '''Job memory limit''' is not relevant (all jobs currently request the default memory/core of 4GB). 
    143145 
    144 Navigate to ''model selection-> compile and modifications-> UM user override files'' and include /home/grenville/umui_jobs/overrides/polaris_7.3_machine and /home/grenville/umui_jobs/overrides/polaris_7.3_file in the tables User machine overrides and User file overrides respectively. 
    145  
    146 '''Gotchas for UKCA @ vn 7.3''' There are several places in the code where hard wired absolute paths are used, namely, in the routines ukca_read_aerosol.F90 and ukca_phot2d.F90. We have merely patched the code to point to files on Polaris. 
    147  
    148 == 4 Example UMUI Jobs == 
     146Navigate to ''model selection-> compile and modifications-> UM user override files'' and include `/home/grenville/umui_jobs/overrides/polaris_7.3_machine` and `/home/grenville/umui_jobs/overrides/polaris_7.3_file` in the tables User machine overrides and User file overrides respectively. 
     147 
     148'''Gotchas for UKCA @ vn 7.3''' \\ 
     149There are several places in the code where hard wired absolute paths are used, namely, in the routines `ukca_read_aerosol.F90` and `ukca_phot2d.F90`. We have merely patched the code to point to files on Polaris. 
     150 
     151== 4. Example UMUI Jobs == 
    149152The following jobs are currently under user grenville in the UMUI 
    150153||Job Id||  Model  || 
     
    152155||xideh||UKCA-!StratChem N48 L60 QESM-A|| 
    153156 
     157---- 
    154158 
    155159= Setting up UM 8.4 to run on Polaris = #vn8.4 
    156160 
    157 == 1 Central Installation Directory Structure == 
    158 The vn 8.4 installation is under /home/polaris_lds1/earhum/vn8.4/.  
    159  
    160  
    161 == 2 Your Setup on Polaris == 
    162 Copy the UM Setup section from /home/polaris_lds1/ldsgl/.bashrc to your .bashrc. In order to use vn 8.4 utilities make sure you source .umsetvars_8.4. 
    163  
    164 == 3 Model Build Information and UMUI Settings == 
     161== 1. Central Installation Directory Structure == 
     162The vn 8.4 installation is under `/home/polaris_lds1/earhum/vn8.4`.  
     163 
     164 
     165== 2. Your Setup on Polaris == 
     166Copy the UM Setup section from `/home/polaris_lds1/ldsgl/.bashrc` to your `.bashrc`. In order to use vn 8.4 utilities make sure you source `.umsetvars_8.4`. 
     167 
     168== 3. Model Build Information and UMUI Settings == 
    165169The model build information is kept in configuration files which are read by the FCM build system. The appropriate compiler flags are used to build the model along with information about which libraries to link. We have used the following flags for the Polaris intel compiler for both model and communications builds 
    166170