Changes between Version 27 and Version 28 of Docs/Polaris


Ignore:
Timestamp:
08/02/16 16:16:33 (4 years ago)
Author:
ros
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Docs/Polaris

    v27 v28  
    77CMS have installed several UM versions on Polaris.  Details of how to setup and run on Polaris can be found below: 
    88 
    9  * UM 6.6.3 
     9 * [wiki:/Docs/Polaris/UM6.6.3 UM 6.6.3] 
    1010 
    11  * UM7.3 
     11 * [wiki:/Docs/Polaris/UM7.3 UM 7.3] 
    1212 
    13  * UM 8.4 
    14  
    15 = Setting up UM 6.6.3 to run on Polaris = 
    16  
    17 == 1. Central Installation Directory Structure == 
    18 The central UM directory is `/home/polaris_lds1/earhum`. FCM (code management system), gcom (UM communications software), and UM vn 6.6.3 are held in this directory. Other versions of the UM and related software will also be installed here.  
    19  
    20 The central installation of UM vn 6.6.3 is in `/home/polaris_lds1/earhum/hg6.6.3` 
    21 ||'''Directory'''||'''Description of Contents'''|| 
    22 ||HG2ES_ancils||Ancillary files for the HadGEM2-ES model|| 
    23 ||HG2CCL60_ancils||Ancillary files for the HadGEM2-CC L60 model and example start files|| 
    24 ||HG2AO_ancils||Ancillary files for the HadGEM2 coupled Atmosphere-Ocean model and example start files|| 
    25 ||HG2AMIP_ancils||Ancillary files for the HadGem2 AMIP model and an example start file|| 
    26 ||dumps||example start dumps for the HadGEM2-ES model|| 
    27 ||ctldata||STASHmaster, ANCILmaster, spectral data, and vertical-levels files|| 
    28 ||sgi||small executables and UM utilities, and installation information|| 
    29  
    30 == 2. Environment Variables and Your Files == 
    31 Standard UM environment variables are set thus 
    32 ||'''Environment Variable'''||'''Value'''|| 
    33 || UMDIR ||/home/polaris_lds1/earhum|| 
    34 || WORKDIR ||/nobackup/$USER|| 
    35 || DEVTDIR ||$WORKDIR|| 
    36 ||DATADIR||$WORKDIR|| 
    37 ||TMPDIR||$WORKDIR/tmp/tmp.polaris.$$|| 
     13 * [wiki:/Docs/Polaris/UM8.4 UM 8.4] 
    3814 
    3915 
    40  
    41  
    42 so dumps, diagnostics, lbcs, history files, and intermediate output will by default be sent to `/nobackup/$USER/$RUNID`. The final leave file for a run will be saved in `/home/polaris_lds1/$USER/output`. 
    43  
    44  
    45 '''Start files. ''' 
    46 Example start files for the four HadGEM2 jobs are provided. We have mimicked the file structure as it exists on HECToR, so start files for HadGEM2-ES are in `$UMDIR/hg6.6.3/dumps`, all other models are provided with example start files along with their ancillary files. 
    47  
    48 == 3. Your Setup on Polaris == 
    49 Copy the UM Setup section from `/home/polaris_lds1/ldsgl/.bashrc` to your `.bashrc` (note, if you wish to use utilities for a version of the UM other than 6.6.3, you will need to source the appropriate `.umsetvars` file) 
    50  
    51 == 4. Model Build Information and UMUI Settings == 
    52 The model build information is kept in configuration files which are read by the FCM build system. The appropriate compiler flags are used to build the model along with information about which libraries to link. We have used the following flags for the Polaris intel compiler for both model and communications builds 
    53  
    54 {{{-i8 -r8 -fp-model precise -O1}}} 
    55  
    56 To ensure that these and all other Polaris build settings are picked up by FCM, navigate to ''model selection -> sub-model independent -> FCM configuration ->  FCM configuration variables'' and set the variable '''UM_SVN_BIND''' to  
    57  
    58 {{{fcm:um_br/dev/ros/hg6.6.3_polaris_machine_cfg/src/configs/bindings}}} 
    59  
    60  
    61 === FCM Branches === 
    62  
    63 We have created a branch with code changes to account for slight differences in the way the intel compiler handles some aspects of the UM code (mostly because of its intolerance to multiple declarations of the same variable). Navigate to ''model selection -> sub-model independent -> FCM configuration -> FCM configuration optional modifications'' and include the following branch 
    64  
    65  
    66 {{{fcm:um_br/dev/grenville/hg6.6.3_polaris_fixes}}} 
    67  
    68 in the '''User Modifications''' table 
    69  
    70  
    71  
    72 === General UMUI Settings === 
    73  
    74 Navigate to ''model selection -> sub-model independent -> FCM configuration ->  FCM configuration variables''. 
    75 Experience will help determine the most convenient places where the extracted model should reside. Explicitly set '''UM_ROUTDIR''' to be your user directory on /nobackup, however, note that files left in this directory may be marked for deletion in accordance with Polaris data policy. 
    76  
    77 Navigate to ''model selection -> user information and target machine -> general details'' 
    78 Set '''User-id''' to be your Polaris id. The '''Tic Code''' is not relevant to Polaris. We have not tested end of run email notification. 
    79  
    80 Navigate to ''model selection -> user information and target machine -> target machine'' 
    81 Chose 'other' for the '''Compile, Link and Run the job''' option, and set the '''Other machine name''' to polaris.leeds.ac.uk 
    82  
    83 Navigate to ''model selection -> sub-model independent -> job submission, resources and resubmission pattern'' 
    84 Choose the option ''''qsub' for SGE(SGI)''' for the submission method. The '''Job memory limit''' is not relevant (all jobs currently request the default memory/core of 4GB). 
    85  
    86 '''Gotchas for HadGEM2-AMIP:''' \\ 
    87 At two places in the UMUI, files need to be specified by full paths: navigate to ''model selection -> atmosphere -> ancillary and input data files -> climatologies and potential climatologies -> natural climate forcing'' and specify /home/polaris_lds1/earhum/hg6.6.3/HG2AMIP_ancils as the path for the '''solar forcing'''  and '''volcanic forcing''' files. 
    88  
    89 We have undertaken limited testing of Climate Meaning to find that inclusion of STASH item 262 (section 0) BOUNDARY LAYER CONVECTION FLAG causes a checksum failure. Switch off this stash item when running with Climate Meaning. 
    90  
    91 == 5. Example UMUI Jobs == 
    92 The following jobs are currently under user grenville in the UMUI 
    93 ||Job Id||  Model  || 
    94 ||xidee||HadGEM2-ES|| 
    95 ||xidex||HadGEM2-AMIP|| 
    96 ||xidey||HadGEM2-CC L60|| 
    97 ||xidez||HadGEM2-AO|| 
    98  
    99  
    100 == 6. Performance and Scaling == #Performance 
    101  
    102  
    103 The following figure summarizes results from test running the four HadGEM2 jobs with increasing processor count. The jobs were run for 3 model days, with the exception of HadGEM2-CC was run for 1 model day. The results for total wall clock time for HadGEM2-ES running on HECToR and CURIE (a PRACE machine (Bull)) are also shown. Data points in red refer to the compute time for the Polaris jobs. All models display a lack of scaling when run on 96 processors. 
    104  
    105 [[Image(hg2.png, 800px, center)]] 
    106  
    107 ---- 
    10816 
    10917= Setting up UM 7.3 to run on Polaris =