Changes between Version 28 and Version 29 of Docs/Polaris


Ignore:
Timestamp:
08/02/16 16:21:36 (3 years ago)
Author:
ros
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Docs/Polaris

    v28 v29  
    1515 
    1616 
    17 = Setting up UM 7.3 to run on Polaris = 
    18  
    19 == 1. Central Installation Directory Structure == 
    20 The vn 7.3 installation is under `/home/polaris_lds1/earhum/vn7.3`. The subdirectory HGPKG2 holds all the data files necessary to run the standard HadGEM3 job xidef (the same job on HECToR is xeozb). 
    21 The various other subdirectorties hold control data for the model, and data to run xideh (a copy of xfvgc). 
    22  
    23 == 2. Your Setup on Polaris == 
    24 Copy the UM Setup section from `/home/polaris_lds1/ldsgl/.bashrc` to your `.bashrc` (note, if you wish to use utilities for a version of the UM other than 6.6.3, you will need to source the appropriate .umsetvars file) 
    25  
    26 == 3. Model Build Information and UMUI Settings == 
    27 The model build information is kept in configuration files which are read by the FCM build system. The appropriate compiler flags are used to build the model along with information about which libraries to link. We have used the following flags for the Polaris intel compiler for both model and communications builds 
    28  
    29 {{{-i8 -r8 -fp-model precise -O1}}} 
    30  
    31 UM vn 7.3 requires the inclusion of override files to correctly set compiler and linker options. Navigate to ''model selection-> compile and modifications-> UM user override files'' and include `/home/grenville/umui_jobs/overrides/polaris_7.3_machine` and `/home/grenville/umui_jobs/overrides/polaris_7.3_file` in the tables '''User machine overrides''' and '''User file overrides''' respectively. 
    32  
    33  
    34 === FCM Branches === 
    35  
    36 Navigate to ''model selection -> sub-model independent -> FCM configuration -> FCM configuration optional modifications'' and include the following branch 
    37  
    38  
    39 {{{fcm:um_br/dev/grenville/vn7.3_polaris_ukca/src}}} 
    40  
    41 in the '''User Modifications''' table. The naming of this necessary branch may change in future - this branch contains minor code changes to point to appropriate UKCA data files on Polaris and a script modification to ensure CRUNS behave correctly but was based on Luke's UKCA-CheM branch, hence make sure that the following branch '''does not''' appear in the list otherwise a build error will result: 
    42  
    43 {{{fcm:um_br/dev/lukeVN7.3_UKCA_CheM/src}}}. 
    44  
    45  
    46  
    47 === General UMUI Settings === 
    48  
    49 Navigate to ''model selection -> sub-model independent -> FCM configuration ->  FCM configuration variables''. 
    50 Experience will help determine the most convenient places where the extracted model should reside. Explicitly set '''UM_ROUTDIR''' to be your user directory on `/nobackup`, however, note that files left in this directory may be marked for deletion in accordance with Polaris data policy. 
    51  
    52 Navigate to ''model selection -> user information and target machine -> general details'' 
    53 Set '''User-id''' to be your Polaris id. The '''Tic Code''' is not relevant to Polaris. We have not tested end of run email notification. 
    54  
    55 Navigate to ''model selection -> user information and target machine -> target machine'' 
    56 Chose 'other' for the '''Compile, Link and Run the job''' option, and set the '''Other machine name''' to `polaris.leeds.ac.uk` 
    57  
    58 Navigate to ''model selection -> input/output control and resources -> job submission, resources and resubmission pattern'' 
    59 Choose the option ''''qsub' for SGE(SGI)''' for the submission method. The '''Job memory limit''' is not relevant (all jobs currently request the default memory/core of 4GB). 
    60  
    61 Navigate to ''model selection-> compile and modifications-> UM user override files'' and include `/home/grenville/umui_jobs/overrides/polaris_7.3_machine` and `/home/grenville/umui_jobs/overrides/polaris_7.3_file` in the tables User machine overrides and User file overrides respectively. 
    62  
    63 '''Gotchas for UKCA @ vn 7.3''' \\ 
    64 There are several places in the code where hard wired absolute paths are used, namely, in the routines `ukca_read_aerosol.F90` and `ukca_phot2d.F90`. We have merely patched the code to point to files on Polaris. 
    65  
    66 == 4. Example UMUI Jobs == 
    67 The following jobs are currently under user grenville in the UMUI 
    68 ||Job Id||  Model  || 
    69 ||xidef||HadGEM3-A r2.0 N96 L85|| 
    70 ||xideh||UKCA-!StratChem N48 L60 QESM-A|| 
    71  
    72 ---- 
    73  
    74 = Setting up UM 8.4 to run on Polaris = #vn8.4 
    75  
    76 == 1. Central Installation Directory Structure == 
    77 The vn 8.4 installation is under `/home/polaris_lds1/earhum/vn8.4`.  
    78  
    79  
    80 == 2. Your Setup on Polaris == 
    81 Copy the UM Setup section from `/home/polaris_lds1/ldsgl/.bashrc` to your `.bashrc`. In order to use vn 8.4 utilities make sure you source `.umsetvars_8.4`. 
    82  
    83 == 3. Model Build Information and UMUI Settings == 
    84 The model build information is kept in configuration files which are read by the FCM build system. The appropriate compiler flags are used to build the model along with information about which libraries to link. We have used the following flags for the Polaris intel compiler for both model and communications builds 
    85  
    86 {{{-i8 -r8 -fp-model precise -O1 -openmp}}} 
    87  
    88  
    89  
    90  
    91 === FCM Branches === 
    92  
    93 Navigate to ''model selection -> FCM configuration -> FCM options for atmosphere and reconfiguration''. 
    94  
    95 Until we introduce keywords, specify revision number 11979 for the UM code base and include the following branch 
    96  
    97  
    98 {{{fcm:um_br/dev/grenville/vn8.4_polaris/src}}} 
    99  
    100 in the '''User Modifications''' table. The naming of this necessary branch may change in future - this branch contains a script change needed to ensure CRUNS behave correctly. 
    101  
    102  
    103  
    104 === General UMUI Settings === 
    105  
    106 Navigate to ''model selection -> FCM configuration ->  FCM extract directories and output levels ''. 
    107 Experience will help determine the most convenient places where the extracted model should reside. Explicitly set '''UM_ROUTDIR''' to be your user directory on /nobackup, however, note that files left in this directory may be marked for deletion in accordance with Polaris data policy. 
    108  
    109 Navigate to ''model selection -> user information and submit method -> general details'' 
    110 Set '''User-id''' to be your Polaris id. The '''Tic Code''' is not relevant to Polaris. We have not tested end of run email notification. 
    111  
    112 Navigate to ''model selection -> user information and target machine -> job submission method'' 
    113 Choose the option ''''qsub' for SGE(SGI)''' for the submission method, enter polaris.leeds.ac.uk for the '''Host name''', select '''Change machine config file''' and enter '''sgi-intel-polaris'''.  
    114  
    115 Navigate to ''model selection -> input/output control and resources -> user hand edit files'' and include '''/home/grenville/umui_jobs/hand_edits/polaris_8.4.1''' in the Hand edits list. This is currently necessary to mange the way Polaris handles the module command in the k shell and selects the mvapich2 module to ensure that IO servers function correctly. Future work should see the need for this to disappear. 
    116  
    117  
    118  
    119 == 4 Example UMUI Job == 
    120 The following job is currently under user grenville in the UMUI 
    121 ||Job Id||  Model  || 
    122 ||xidem||HadGEM3-GA4.0 Polaris|| 
    123  
    124 This job uses the UM I/O servers which require that the model run with at least two OpenMP threads. We have successfully run the model using mvapich2 as the mpi implementation which supports multi-threaded mpi. The model runs with single threading with both openmpi and intemlpi mpi implemantations - in these cases OpenMP code directives are obeyed but the model can not support I/O servers.  
    125  
    126  
    127  
    128