Changes between Version 12 and Version 13 of UM/Configurations/UKESM/RelNotes1.0


Ignore:
Timestamp:
28/03/19 14:39:18 (7 months ago)
Author:
jwalton
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • UM/Configurations/UKESM/RelNotes1.0

    v12 v13  
    44These are the notes for the first release of the UK Earth System Model (UKESM1).  A brief introduction to UKESM1 can be found [wiki:UKESM/UKESM1intro here]. 
    55 
    6 == Model and suite specifications 
     6== Model configurations 
    77 
    8 The current version of UKESM1 has an atmospheric resolution of N96 (~140 km) and a one degree resolution in the ocean.  The vertical resolution is 85 levels in the atmosphere and 75 levels in the ocean.  
     8The model is available in a number of configurations: 
    99 
    10 The model is distributed and run as a [wiki:RoseCylc Rose] suite.  A number of different configurations of the model are available, described in the following subsections.   
     10  * an atmosphere-only (so-called '''AMIP''') configuration, in which the model atmosphere is forced by observed sea surface temperature and sea ice boundary conditions. 
     11  * two fully coupled configurations, which each make use of all model components: 
     12     * one set up to run the CMIP6 '''pre-industrial control''' experiment, and 
     13     * one to run the CMIP6 '''historical''' experiment. 
    1114 
    12 ''Note that links to suites (and to a couple of other pages) require access to the Met Office Science Repository Service (MOSRS) - see [wiki:UKESM/UKESM1intro#Backgroundandprerequisites here] for more details''. 
     15In addition, additional coupled configurations for the CMIP6 '''abrupt4xCO2''' and '''1%CO2''' experiments can be created using the pre-industrial control experiment as a starting point.  
    1316 
    14 === historical, pre-industrial control 
    15  
    16 There are two fully coupled UKESM1 configurations which each make use of all model components: one with science settings for a historical experiment, and one with settings for a pre-industrial control experiment.    
    17   
    18 ||=''UM version''=||=''historical''=||=''pre-industrial control''=|| 
    19 ||=vn11.2=||=[https://code.metoffice.gov.uk/trac/roses-u/browser/b/c/6/1/3/trunk u-bc613]=||=[https://code.metoffice.gov.uk/trac/roses-u/browser/b/c/9/6/4/trunk u-bc964]=|| 
    20  
    21 === abrupt4xCO2, 1%CO2 
    22  
    23 Configurations for abrupt4xCO2 and 1%CO2 experiments can be created using the pre-industrial control experiment as a starting point. 
    24  
    25 To create a suite for the abrupt4xCO2 experiment, first make a copy of the  [https://code.metoffice.gov.uk/trac/roses-u/browser/b/c/9/6/4/trunk pre-industrial control suite], then in `um -> namelist -> UM Science Settings -> Section 01-02 - Radation -> GAS MMRs`, set `co2_mmr=1.72728e-03`. 
    26  
    27 To create a suite for the 1xCO2 experiment, first make a copy of the  [https://code.metoffice.gov.uk/trac/roses-u/browser/b/c/9/6/4/trunk pre-industrial control suite], then in `um -> namelist -> UM Science Settings -> Section 01-02 - Radation -> GAS MMRs -> Varying gas MMRs`, set `l_clmchfcg=.true.` to enable time-varying GHGs.  Finally, in the sub-panel `Varying CO2 MMRs`, set 
    28  
    29 {{{ 
    30 clim_fcg_levls_co2=4.3182e-04 
    31 clim_fcg_nyears_co2=1 
    32 clim_fcg_rates_co2=1.0 
    33 clim_fcg_years_co2=1849 
    34 }}} 
    35  
    36  
    37 See [#Sciencenotes below] for more on the science settings of the model. 
    38  
    39 == Options for running the model 
    40  
    41 By default, each UKESM1 suite is set up to run the model on the Met Office HPC.  The suite offers several options for specifying how the model is to be run, including:   
    42  
    43  * login node to be used for submission to Met Office HPC: `suite conf -> Machine Options -> MetO Cray login node` 
    44  * Met Office queue to which jobs will be submitted: `suite conf -> Machine Options -> HPC queue`   
    45  
    46 Options for specifying the account under which jobs will be run are available in `suite conf -> Project Accounting`: 
    47  
    48  * Select `Use default account` to use the default account for your department.   
    49  * If this is set to `false`, then choose an option from the `Account` menu. 
    50  * If the option is `other`, then enter the account explicitly in `'Other' user account`. 
    51  
    52 === Running on other machines 
    53  
    54 The model may be run on other (i.e. non-Met Office) machines.  See [wiki:UKESM/UKESM1intro#Backgroundandprerequisites here] for more on available resources and how to access them.  More specific instructions for suite settings for different machines are given in the following subsections. 
    55  
    56 ==== Monsoon 
    57  
    58 To run on Monsoon, the Met Office / NERC collaborative platform, set `suite conf -> Machine Options -> Site at which model is being run` to  `MONSooN`.   
    59  
    60 Output files created by the suite running on Monsoon may be archived via the Met Office Operational Storage Environment (MOOSE).  The options for requesting this can be found under the `postproc -> Post Processing - common settings` control panel.  Set `archive_command` to `Moose` and provide (or check) values for further options in the subpanel `Moose Archiving`.  See [#Archivingofduplexeddata below] for more on the `non_duplexed_set` option.  
    61  
    62 Note that you must have a MOOSE account before archiving can work - see [#Support below] for help.  
    63  
    64 ==== NEXCS 
    65  
    66 To use NEXCS, the NERC-only share of Monsoon, set `suite conf -> Machine Options -> Site at which model is being run` to  `MONSooN` and select a NEXCS account for running jobs (see [#Optionsforrunningthemodel above]).  
    67  
    68 Output files created by the suite running on NEXCS may be archived to disk.  The options for requesting this can be found under the `postproc -> Post Processing - common settings` control panel.  Set `archive_command` to `NEXCS` and provide values for `archive_root_path` and `archive_name` in the subpanel `Archer Archiving` (sic) to specify the location of the archived files on NEXCS.   
    69  
    70 Following archiving, the files may be optionally transferred to a remote machine such as JASMIN.  Provide values for `remote_host` (the address of the remote machine) and `transfer_dir` (the location of the archived files on the remote machine) in the subpanel `JASMIN Transfer`.  In addition, transferring must be turned on by setting `suite conf -> Build and Run -> PP Transfer` to `true`. 
    71  
    72 Note that, before transfer from NEXCS to JASMIN can work, some setup of communications is required - see [wiki:Docs/PostProcessingAppNexcsSetup#sshjasmin here] for more details. 
    73  
    74 ==== Archer  
    75  
    76 To run on Archer, the NERC platform, set `suite conf -> Machine Options -> Site at which model is being run` to  `Archer` and set these other `Machine Options`: 
    77  
    78   * `Use Environment Modules` to `Custom module files`  
    79   * `Science Configuration Module Name` to `GC3-PrgEnv/2.0/90386` 
    80   * `Module file location` to `/work/y07/y07/umshared/moci/modules/modules` 
    81  
    82 In addition, the following tests (see [#Testsinthesuite below]) ''must'' be turned off when running on Archer: 
    83    
    84   * `Test restartability` 
    85   * `Test rigorous compiler option` 
    86   * `Test PE decomposition change`  
    87   * `Archive integrity` 
    88   * `CPMIP Analysis -> CPMIP load balancing analysis` 
    89  
    90 Output files created by the suite running on Archer may be archived to disk.  The options for requesting this can be found under the `postproc -> Post Processing - common settings` control panel.  Set `archive_command` to `Archer` and provide values for `archive_root_path` and `archive_name` in the subpanel `Archer Archiving` to specify the location of the archived files on Archer.   
    91  
    92 Following archiving, the files may be optionally transferred to a remote machine such as JASMIN.  Provide values for `remote_host` (the address of the remote machine) and `transfer_dir` (the location of the archived files on the remote machine) in the subpanel `JASMIN Transfer`.  In addition, transferring must be turned on by setting `suite conf -> Build and Run -> PP Transfer` to `true`. 
    93  
    94 Note that, before transfer from Archer to JASMIN can work, some setup of communications (specifically, ''both'' [wiki:Docs/PostProcessingAppArcherSetup#sshpumatodtn between PUMA and Archer data transfer node], ''and'' [wiki:Docs/PostProcessingAppArcherSetup#sshdtntojasmin between Archer data transfer node and JASMIN]) is required. 
    95  
    96 == Using the model in a CMIP6 production run 
    97  
    98 Using the model in a CMIP6 production run requires the provision of extra information (for example, the MIP and the experiment to which the run is contributing).  This information must be specified in the suite, and the suite will check its validity.  See [https://code.metoffice.gov.uk/trac/ukcmip6/wiki/CMIP6/RoseSuiteMetadata here on the MOSRS] for more details about this information. 
    99  
    100 By default, the suite is ''not'' set up to run in full CMIP6 production mode, so this information is not required.  To change this, set `suite info -> project` to `u-cmip6` (which will expose the interface in `suite info` for collecting this information) and set `suite conf -> Project Accounting -> CMIP6 Experiment` to `true` (which will turn on the validation of the information).  More information on setting up a CMIP6 experiment is available [https://code.metoffice.gov.uk/trac/ukcmip6/wiki/ExperimentGuidance here on the MOSRS]. 
    101  
    102 == Archiving of duplexed data 
    103  
    104 When running on Met Office machines (including Monsoon), the suite will, by default, archive ''a single copy'' of its data to MOOSE.  For critical model runs, this setting may be changed to archive two copies of the data (i.e. duplex) by switching `non_duplexed_set` in `postproc -> Post Processing-common settings -> Moose Archiving` to `false`.  Further guidance on when to choose this option is available at http://www-twiki/Main/MassNonDuplexPolicy (note that this link only works from within the Met Office). 
    105  
    106 == Compute resource usage 
    107  
    108 The compute resources used by the suite can be set via parameters on the `Machine Options` and `Domain Decomposition` control panels under `suite conf`.  The following discussion is specific to the Met Office HPC for the most part, but may still be helpful for users of other machines. 
    109  
    110 The type of compute node can be set via `suite conf -> Machine Options -> XC40 core type`: a `Haswell` node has 32 cores, while `Broadwell` has 36.   
    111  
    112 The suite is currently set up in `suite conf -> Domain Decomposition` to use 36 nodes (see [#Calculationofnodecount below] for more details on how this is calculated).  An alternative setup uses 19 nodes.  Parameter settings for both setups are: 
    113  
    114 ||=''Parameter''=||=''36 node suite''=||=''19 node suite''=|| 
    115 ||=Atmosphere: Processes East-West=||=32=||=32=|| 
    116 ||=Atmosphere: Processes North-South=||=18=||=18=|| 
    117 ||=IO Server Processes=||=0=||=0=|| 
    118 ||=OpenMP threads for the atmosphere=||=2=||=1=|| 
    119 ||=NEMO: Number of processes East-West=||=12=||=9=|| 
    120 ||=NEMO: Number of processes North-South=||=9=||=8=|| 
    121 ||=NEMO: Number of processes in XIOS server=||=6=||=6=|| 
    122 ||=OpenMP threads for the ocean=||=1=||=1=|| 
    123  
    124 Note that the ocean must be rebuilt (by setting `suite conf -> Build and Run -> Build Ocean` to `true`) whenever the NEMO parameters in the table are changed during a run.  
    125  
    126 Setting these parameters to other values may require load balancing to ensure that HPC resources are being used in the most efficient fashion.  
    127  
    128 === Calculation of node count 
    129  
    130 On `Domain Decomposition -> Atmosphere`, the number of processes used by the UM can be set via `Atmosphere: Processes East-West` and `Atmosphere: Processes North-South`; additional processes for the IO Server may be requested using `IO Server Processes`.  Finally, `OpenMP threads for the atmosphere` sets the number of threads for each process; multiplying this by the number of processes gives the number of compute tasks.   
    131  
    132 Using the parameter values for the ''36 node suite'', the number of tasks used by the UM is `(32 * 18 + 0) * 2 = 1152`. Dividing by the number of cores per node (in this case `36`) and rounding up (because different executables cannot run on the same node) gives `32` compute nodes used by the atmosphere. 
    133  
    134 A similar calculation may be performed for the settings on `Domain Decomposition -> Ocean` using `NEMO: Number of processes East-West`, `NEMO: Number of processes North-South` and `OpenMP threads for the ocean` to give `12 * 9 * 1 = 108` tasks, or `3` compute nodes used by the ocean. 
    135  
    136 Finally, on the same control panel, `NEMO: Number of processes in XIOS server` is set to `6`, which equates to `1` compute node used by XIOS.   
    137  
    138 Thus, the total number of nodes used by the suite is `32 + 3 + 1 = 36`. 
    139  
    140 == Tests in the suite 
    141  
    142 The suite contains options for testing different aspects of the model including reproducible restarting, changes in processor decomposition, comparison to known good output and integrity of archived files.  Some of these tests may be of more interest to developers than general users of the model; they can be turned on or off via the `suite conf -> Testing` control panel. 
    143  
    144 === Testing for PE decomposition change 
    145  
    146 Changing the PE decomposition will change the results of the model because of the behaviour of the chemistry solver within the UM. Thus, by default, the `Test PE decomposition change` test (see `suite conf -> Testing`) will fail, and so this test has been turned off. There is a version of the chemistry solver which does not change results; this can be selected by setting `l_ukca_asad_columns=.true.` in `app/um/rose-app.conf`. With this option selected, the PE decomposition change test should pass.  
    147  
    148 Note that we do not select this version of the chemistry solver by default because it has a performance overhead; specifically, it causes an atmosphere-only job to run about 10% slower than when running with `l_ukca_asad_columns=.false.`. 
    149  
    150 It should be noted that changing the PE decomposition for the ocean in UKESM will also change results because this changes results for both the iceberg code in NEMO and for the CICE code. This behaviour cannot be rectified by setting a single variable (that is, there is no analogue of `l_ukca_asad_columns` for NEMO and CICE).  
    151  
    152 == Science notes 
    153  
    154 The historical release job differs from the first member of the CMIP6 historical ensemble (`u-bb075`) in the following ways: 
    155 * The anthropogenic SO2 emission ancillaries were produced using different methodologies. The resulting SO2 emission in the model is nearly identical but has differences at the bit level. 
    156 * It includes a fix for the aerosol plume scavenging diagnostic 38900-38932 to ensure that they bit-compare across differing processor decompositions. The difference appears at only a small number of points where the value is very close to zero. 
    157  
    158 The piControl release job differs from the CMIP6 piControl run (`u-aw310`) in the following ways: 
    159 * The anthropogenic emission height is different. In the release job all anthropogenic SO2 is emitted at the surface, consistent with the model's treatment of other anthropogenic emissions (except NOx emissions from aircraft which are supplied with a vertical emission profile). In the CMIP6 piControl, around 30% of anthropogenic emissions were released at 500m. Because the 1850 anthropogenic emissions are tiny relative to natural emissions from volcanoes and the ocean, this difference in emission height makes no meaningful difference to the SO2 burden or the aerosol simulation. 
    160 * The release job includes several diagnostic fixes which were not included in the CMIP6 piControl run. These include: 
    161    a. The fix to plume scavenging diagnostics described above 
    162    b. Corrections to diagnostics 30312, 30313, 30298 which were corrupted in the CMIP6 piControl 
    163    c. Addition of some other diagnostics to CMIP6 PP streams 
    164  
    165 == Known issues 
    166  
    167 (none) 
     17A full description of how to access and run the AMIP configuration is [wiki:UKESM/RelNotes1.0/AMIP here], whilst [wiki:UKESM/RelNotes1.0/Coupled this page] contains the instructions for accessing and running the coupled configurations. 
    16818 
    16919== Support