wiki:Archer

Version 1 (modified by annette, 4 years ago) (diff)

ARCHER

ARCHER is the UK's high-end computing service, funded by EPSRC and NERC. It is available for use by academia and industry in the UK and Europe.

ARCHER is a Cray XC30 machine consisting of 4920 compute nodes. Each node is made up of two 2.7 GHz, 12-core Ivy Bridge processors, and either 64 or 128 GB of memory. This provides a total of 118,080 cores in the system. Nodes are connected through Cray's Aries interconnect technology. ARCHER also provides a serial queue for compilation and other tasks. The file systems are split into:

  • /home: A small backed-up space.
  • /work: A larger region optimised for fast access from the compute nodes. It is intended for temporaray storage of simulation, and is not backed-up.
  • /nerc: The Research Data Facility archive for long term storage. The RDF also has a dedicated processing system - the Data Analytic Cluster (DAC).

For more information about the ARCHER service see the webpage: http://www.archer.ac.uk

To get an n02 account on ARCHER, contact grenville.lister AT ncas.ac.uk.

The UM on ARCHER

UM versions installed on ARCHER: 6.1, 6.6.3, 7.3, 7.8, 8.2, 8.4, 8.5, and 8.6. This means that you can run umui jobs on ARCHER for these versions of the model. However, we are still testing various model configurations; results of our tests can be found here.

Setting up UMUI jobs for ARCHER

Things to do on PUMA

Copy a HECToR UMUI job, name it appropriately, then set the machine name to login.archer.ac.uk - different versions of the model use for "machine name" various phrases; Host name in UM 8.5 for example or Other machine name in UM 6.6.3.

ARCHER has 24 processors(cores) per node, so in order to use whole nodes (your job reserves whole nodes even if the number of cores required is not a multiple of 24), you should ensure that you use a multiple of 24 processors (a HECToR node had 32 processors). ARCHER has significantly more memory per processor than did HECToR, so if you were running with a non default number of cores per node, it's worth experimenting with this.

Version specifics

For UM version 7.3 set the User machine overrides file to be

 ~umui/overrides/archer_cce_7.3_machine

and the User file overrides file to be

~umui/overrides/archer_cce_7.3_file

Things to do on ARCHER

Set up ssh keys - you can simply copy the authorized_keys file from HECToR to ARCHER.

Ensure that your .profile is appropriate on ARCHER, again, this can probably just be copied directly from HECToR. Ensure the following lines are in your ARCER profile

. /etc/profile
. /etc/bash.bashrc

Note that the environment variable TARGET_MC is set to cce on HECToR - this tells the UM build system to use the Cray compiler. On ARCHER we have again adopted the Cray compiler and TARGET_MC=cce remains the correct choice on ARCHER, however, we are experimenting with the Intel compiler. Initial tests indicate that there is little to choose between Intel and Cray.

UM Performance and domain-decomposition considerations

A quick scan of our test results indicates that jobs running on ARCHER with the same domain decomposition as on HECToR will run in half the wall clock time. However, the AU cost of an ARCHER core-hr is 15AU whereas a HECToR core-hr was 4AU, thus maintaining the same processor count will result in a cost increase by a factor of 15/8 , ie nearly twice as AU-costly on ARCHER as on HECToR. This represents good value since the NERC AU award is nearly 15/4 times bigger than on HECToR. However, it is well worthwhile experimenting with processor decomposition to find an efficient configuration.

UM Archiving on ARCHER

Instructions for setting up automatic archiving of UM output can be found here.

Porting a job from MONSooN to Archer

Instructions can be found here.

Interactive UM debugging with DDT

Instructions can be found here.

Running the SCM on the ARCHER post processor

Instructions can be found here.

Upgrading to cce8.3.7

Notes and instructions here.