Setting up UM 7.3 to run on Polaris
1. Central Installation Directory Structure
The vn 7.3 installation is under /home/polaris_lds1/earhum/vn7.3. The subdirectory HGPKG2 holds all the data files necessary to run the standard HadGEM3 job xidef (the same job on HECToR is xeozb). The various other subdirectorties hold control data for the model, and data to run xideh (a copy of xfvgc).
2. Your Setup on Polaris
Copy the UM Setup section from /home/polaris_lds1/ldsgl/.bashrc to your .bashrc (note, if you wish to use utilities for a version of the UM other than 6.6.3, you will need to source the appropriate .umsetvars file)
3. Model Build Information and UMUI Settings
The model build information is kept in configuration files which are read by the FCM build system. The appropriate compiler flags are used to build the model along with information about which libraries to link. We have used the following flags for the Polaris intel compiler for both model and communications builds
-i8 -r8 -fp-model precise -O1
UM vn 7.3 requires the inclusion of override files to correctly set compiler and linker options. Navigate to model selection→ compile and modifications→ UM user override files and include /home/grenville/umui_jobs/overrides/polaris_7.3_machine and /home/grenville/umui_jobs/overrides/polaris_7.3_file in the tables User machine overrides and User file overrides respectively.
Navigate to model selection → sub-model independent → FCM configuration → FCM configuration optional modifications and include the following branch
in the User Modifications table. The naming of this necessary branch may change in future - this branch contains minor code changes to point to appropriate UKCA data files on Polaris and a script modification to ensure CRUNS behave correctly but was based on Luke's UKCA-CheM branch, hence make sure that the following branch does not appear in the list otherwise a build error will result:
General UMUI Settings
Navigate to model selection → sub-model independent → FCM configuration → FCM configuration variables. Experience will help determine the most convenient places where the extracted model should reside. Explicitly set UM_ROUTDIR to be your user directory on /nobackup, however, note that files left in this directory may be marked for deletion in accordance with Polaris data policy.
Navigate to model selection → user information and target machine → general details Set User-id to be your Polaris id. The Tic Code is not relevant to Polaris. We have not tested end of run email notification.
Navigate to model selection → user information and target machine → target machine Chose 'other' for the Compile, Link and Run the job option, and set the Other machine name to polaris.leeds.ac.uk
Navigate to model selection → input/output control and resources → job submission, resources and resubmission pattern Choose the option 'qsub' for SGE(SGI) for the submission method. The Job memory limit is not relevant (all jobs currently request the default memory/core of 4GB).
Navigate to model selection→ compile and modifications→ UM user override files and include /home/grenville/umui_jobs/overrides/polaris_7.3_machine and /home/grenville/umui_jobs/overrides/polaris_7.3_file in the tables User machine overrides and User file overrides respectively.
Gotchas for UKCA @ vn 7.3
There are several places in the code where hard wired absolute paths are used, namely, in the routines ukca_read_aerosol.F90 and ukca_phot2d.F90. We have merely patched the code to point to files on Polaris.
4. Example UMUI Jobs
The following jobs are currently under user grenville in the UMUI
|xidef||HadGEM3-A r2.0 N96 L85|
|xideh||UKCA-StratChem N48 L60 QESM-A|