wiki:Archer/NercArchiving

UM Archiving on Archer

1. Introduction

A method for automatic archiving of UM output on the national HPC facility has been adapted from the existing UM archiving code used by the Met Office. The archiving system can archive standard UM output files including climate means, restart dumps and boundary data files. Archiving to the RDF (/nerc) is currently only available for UM versions 6.6.3, 7.3, 8.4, 8.5 and 8.6.

2. Methodology

On Archer, when the UM has finished reading/writing files to be archived, it copies them from $DATAM (defined in the UMUI) to some other directory of your choice. Specifying /nerc/... as the target archive directory moves data from the Lustre work file space to the GPFS Research Data Facility.

3. UMUI changes for Automatic Archiving

  • You must include the appropriate branch when building the UM executable.
    Navigate to model selection → FCM configuration → FCM options for atmosphere and reconfiguration and add the archiving branch for UM version you are using.

For UM version 6.6.3:
Add the branch fcm:um_br/dev/jeff/HG6.6.3_hector_monsoon_archiving/src in the User Modifications table.

For UM version 7.3:
Add the branch fcm:um_br/dev/jeff/VN7.3_hector_monsoon_archiving/src in the User Modifications table.

For the 7.3 version of HadGEM3-A_r2.0 a different branch must be used, namely fcm:um_br/dev/jeff/VN7.3_HadGEM3-A_r2.0_hector_monsoon_archiving/src. This branch is a replacement for the standard HadGEM3-A_r2.0 branch fcm:um_br/pkg/Share/VN7.3_HadGEM3-A_r2.0/src.

For UM version 8.4:
Add the branch fcm:um_br/dev/jeff/vn8.4_hector_monsoon_archiving/src in the Central Script Modifications section.

For UM version 8.5:
Add the branch fcm:um_br/dev/jeff/vn8.5_hector_monsoon_archiving/src in the Central Script Modifications section.

For UM version 8.6:
Add the branch fcm:um_br/dev/jeff/vn8.6_archer_monsoon_archiving/src in the Central Script Modifications section.

  • Navigate to the UMUI window input/output control and resources → user hand-edit files and include the appropriate hand edit (for vn 7.3 use ~jeff/umui_jobs/hand_edits/archiving_7.3). UM versions 8.4 and 8.5 do not require the use of hand edits.
  • Navigate to model selection → post processing → main switch. Select automatic post processing. By checking Delete superseded […], the relevant file will be removed from the work space after being archived, otherwise it will simply be copied to the archive. Select HECToR archive, select Disk, and enter the directory where you want the files to go - typically /nerc/n02/n02/"your-username"/"archive directory".
  • Navigate to model selection → post processing → initialization and processing of mean & standard PP files and enter "Y" for streams to be archived in the column labelled Archive.

4. Set up ssh keys

To allow the UM to archive data to the /nerc disk on Archer the following commands need to be done once on Archer

mkdir -p ~/.ssh

cd ~/.ssh

ssh-keygen -f um_arch

cat um_arch.pub >> authorized_keys

As passphraseless access is required, in response to the request for a passphrase from ssh-keygen hit enter twice to generate an empty passphrase.

Last modified 3 years ago Last modified on 14/04/14 15:43:49