| 62 | | * `Use Environment Modules` to `Custom module files` |
| 63 | | * `Science Configuration Module Name` to `GC3-PrgEnv/2.0/90386` |
| 64 | | * `Module file location` to `/work/y07/y07/umshared/moci/modules/modules` |
| 65 | | |
| 66 | | In addition, the following tests (see [#Testsinthesuite below]) ''must'' be turned off when running on Archer: |
| 67 | | |
| 68 | | * `Test restartability` |
| 69 | | * `Test rigorous compiler option` |
| 70 | | * `Test PE decomposition change` |
| 71 | | * `Archive integrity` |
| 72 | | * `CPMIP Analysis -> CPMIP load balancing analysis` |
| | 62 | In addition, the maximum number of processes per node (see `suite conf -> Domain Decomposition -> Atmosphere -> Max number of processes/node`) must be set to `24`. This value must also be set for the `Max number of process/node` parameter in `suite conf -> Testing -> Processor Decomposition` and `suite conf -> Testing -> OpenMP` if the respective tests have been turned on (see [#Testsinthesuite below]). |
| 90 | | == Compute resource usage |
| 91 | | |
| 92 | | The compute resources used by the suite can be set via parameters on the `Machine Options` and `Domain Decomposition` control panels under `suite conf`. The following discussion is specific to the Met Office HPC for the most part, but may still be helpful for users of other machines. |
| 93 | | |
| 94 | | The type of compute node can be set via `suite conf -> Machine Options -> XC40 core type`: a `Haswell` node has 32 cores, while `Broadwell` has 36. |
| 95 | | |
| 96 | | The suite is currently set up in `suite conf -> Domain Decomposition` to use 36 nodes (see [#Calculationofnodecount below] for more details on how this is calculated). An alternative setup uses 19 nodes. Parameter settings for both setups are: |
| 97 | | |
| 98 | | ||=''Parameter''=||=''36 node suite''=||=''19 node suite''=|| |
| 99 | | ||=Atmosphere: Processes East-West=||=32=||=32=|| |
| 100 | | ||=Atmosphere: Processes North-South=||=18=||=18=|| |
| 101 | | ||=IO Server Processes=||=0=||=0=|| |
| 102 | | ||=OpenMP threads for the atmosphere=||=2=||=1=|| |
| 103 | | ||=NEMO: Number of processes East-West=||=12=||=9=|| |
| 104 | | ||=NEMO: Number of processes North-South=||=9=||=8=|| |
| 105 | | ||=NEMO: Number of processes in XIOS server=||=6=||=6=|| |
| 106 | | ||=OpenMP threads for the ocean=||=1=||=1=|| |
| 107 | | |
| 108 | | Note that the ocean must be rebuilt (by setting `suite conf -> Build and Run -> Build Ocean` to `true`) whenever the NEMO parameters in the table are changed during a run. |
| 109 | | |
| 110 | | Setting these parameters to other values may require load balancing to ensure that HPC resources are being used in the most efficient fashion. |
| 111 | | |
| 112 | | === Calculation of node count |
| 113 | | |
| 114 | | On `Domain Decomposition -> Atmosphere`, the number of processes used by the UM can be set via `Atmosphere: Processes East-West` and `Atmosphere: Processes North-South`; additional processes for the IO Server may be requested using `IO Server Processes`. Finally, `OpenMP threads for the atmosphere` sets the number of threads for each process; multiplying this by the number of processes gives the number of compute tasks. |
| 115 | | |
| 116 | | Using the parameter values for the ''36 node suite'', the number of tasks used by the UM is `(32 * 18 + 0) * 2 = 1152`. Dividing by the number of cores per node (in this case `36`) and rounding up (because different executables cannot run on the same node) gives `32` compute nodes used by the atmosphere. |
| 117 | | |
| 118 | | A similar calculation may be performed for the settings on `Domain Decomposition -> Ocean` using `NEMO: Number of processes East-West`, `NEMO: Number of processes North-South` and `OpenMP threads for the ocean` to give `12 * 9 * 1 = 108` tasks, or `3` compute nodes used by the ocean. |
| 119 | | |
| 120 | | Finally, on the same control panel, `NEMO: Number of processes in XIOS server` is set to `6`, which equates to `1` compute node used by XIOS. |
| 121 | | |
| 122 | | Thus, the total number of nodes used by the suite is `32 + 3 + 1 = 36`. |
| 123 | | |
| 128 | | === Testing for PE decomposition change |
| 129 | | |
| 130 | | Changing the PE decomposition will change the results of the model because of the behaviour of the chemistry solver within the UM. Thus, by default, the `Test PE decomposition change` test (see `suite conf -> Testing`) will fail, and so this test has been turned off. There is a version of the chemistry solver which does not change results; this can be selected by setting `l_ukca_asad_columns=.true.` in `app/um/rose-app.conf`. With this option selected, the PE decomposition change test should pass. |
| 131 | | |
| 132 | | Note that we do not select this version of the chemistry solver by default because it has a performance overhead; specifically, it causes an atmosphere-only job to run about 10% slower than when running with `l_ukca_asad_columns=.false.`. |
| 133 | | |
| 134 | | It should be noted that changing the PE decomposition for the ocean in UKESM will also change results because this changes results for both the iceberg code in NEMO and for the CICE code. This behaviour cannot be rectified by setting a single variable (that is, there is no analogue of `l_ukca_asad_columns` for NEMO and CICE). |
| 135 | | |
| 138 | | The historical release job differs from the first member of the CMIP6 historical ensemble (`u-bb075`) in the following ways: |
| 139 | | * The anthropogenic SO2 emission ancillaries were produced using different methodologies. The resulting SO2 emission in the model is nearly identical but has differences at the bit level. |
| 140 | | * It includes a fix for the aerosol plume scavenging diagnostic 38900-38932 to ensure that they bit-compare across differing processor decompositions. The difference appears at only a small number of points where the value is very close to zero. |
| 141 | | |
| 142 | | The piControl release job differs from the CMIP6 piControl run (`u-aw310`) in the following ways: |
| 143 | | * The anthropogenic emission height is different. In the release job all anthropogenic SO2 is emitted at the surface, consistent with the model's treatment of other anthropogenic emissions (except NOx emissions from aircraft which are supplied with a vertical emission profile). In the CMIP6 piControl, around 30% of anthropogenic emissions were released at 500m. Because the 1850 anthropogenic emissions are tiny relative to natural emissions from volcanoes and the ocean, this difference in emission height makes no meaningful difference to the SO2 burden or the aerosol simulation. |
| 144 | | * The release job includes several diagnostic fixes which were not included in the CMIP6 piControl run. These include: |
| 145 | | a. The fix to plume scavenging diagnostics described above |
| 146 | | b. Corrections to diagnostics 30312, 30313, 30298 which were corrupted in the CMIP6 piControl |
| 147 | | c. Addition of some other diagnostics to CMIP6 PP streams |
| | 86 | (none) |
