NCAS Computational Modelling Services

Unified Model

Migration of UM Data to JASMIN Elastic Tape

Instructions on how to add the JDMA app to a UM Suite to enable migration of model data to JASMIN Elastic Tape as part of the UM Workflow.

These instructions assume that your suite is already setup to use the correct version of Postproc. See the Post-Processing page.

Setup and initialise JDMA on JASMIN

Before you can use the JDMA to migrate data to Elastic Tape you must install the jdma client on JASMIN and initialise some user settings.

  1. Install JDMA client software:
    • Login to a JASMIN sci machine
    • Load python2:
      module load jaspy/2.7
    • Create a virtual environment in your home directory:
      virtualenv ~/jdma_venv_py2
    • Activate the virtual environment:
      source ~/jdma_venv_py2/bin/activate
    • Install the jdma client into the virtualenv:
      pip install git+https://github.com/cedadev/jdma_client
  2. Follow instructions in the Setting up the user, user settings and user info section of the JDMA Tutorial.

UM Rose suite changes

Rose Edit

  1. In panel suite conf -> JASMIN Elastic Tape:
    • Set Migrate data to JASMIN Elastic Tape to True
  2. In panel postproc -> Post Processing - common settings -> ARCHER2-JASMIN Archiving -> Elastic Tape:
    • Set default_workspace to False
    • Enter the JASMIN GWS under which you wish to migrate the data to ET (e.g. ncas_climate).

Setup connection to JASMIN sci nodes

  1. Add the following to your ~/.ssh/config file on PUMA:
# JASMIN
Host login2
Hostname login2.jasmin.ac.uk
User <jasmin_username> 
IdentityFile ~/.ssh/<jasmin-ssh-key>
ForwardAgent yes
ControlMaster auto
ControlPath /tmp/ssh-socket-%r@%h-%p
ControlPersist yes

Host sci? cylc1
Hostname %h.jasmin.ac.uk

Host sci* cylc*
User <jasmin_username>
IdentityFile ~/.ssh/<jasmin-ssh-key>
ForwardAgent yes
ProxyCommand ssh -Y login2 -W %h:%p
ControlMaster auto
ControlPath /tmp/ssh-socket-%r@%h-%p
ControlPersist yes

  1. Add your JASMIN ssh-key to your ssh-agent:

    • ssh-add ~/.ssh/<jasmin-ssh-key>

  2. Test connection to JASMIN:

    • ssh sci3.jasmin.ac.uk
    • You should be logged into the JASMIN sci node without prompt for your JASMIN passphrase.

  3. Add path to Rose/Cylc to your ~/.bash_profile on JASMIN:

  if [[ $(hostname) = sci*.jasmin.ac.uk || $(hostname) = cylc*.jasmin.ac.uk ]]; then
    # Rose/cylc on jasmin-sci & Lotus nodes
    export PATH=/apps/jasmin/metomi/bin:$PATH
  fi

Configure PPTransfer

If you haven’t already done so follow the instructions for configuring PPTransfer and setting up Gridftp certificate: Configuring PPTransfer

You may wish to consider using the JASMIN transfer cache disk as the JASMIN transfer destination. This is a large temporary storage area separate to the Group Workspaces. See JASMIN Transfer Cache for more information.