wiki:CDDS

Version 6 (modified by grenville, 6 months ago) (diff)

CMIP6 data CMOR'isation - How to run the CDDS work flow on JASMIN

1. JASMIN GWS access

You will need access to the cmip6_prep Group Workspace. Apply here:

https://accounts.jasmin.ac.uk/services/group_workspaces/cmip6_prep/

2. MOSRS access

You will need MOSRS access from JASMIN, see https://code.metoffice.gov.uk/trac/home/wiki/AuthenticationCaching

3. Where to run

The CDDS work flow comprises several phases

  1. preparation
  2. data reformatting
  3. quality check

Phase A runs interactively, is light on computational resource and can be run on jasmin-cylc. Phase B is computationally intensive, runs on LOTUS, and monitored from jasmin-cylc. Phase C is potentially computationally intensive and should not be run on jasmin-cylc - one of the jasmin sci machines will be appropriate.

4. Model Data

CDDS expects data to be in a JASMIN group workspace. CDDS supports two data directory structures:

  1. data by stream - data from MASS will be structured by stream (ap4, ap5, apm…)
  2. data bu cycle - data from ARCHER or NEXCS will be structured by cylc cycle (18500101T0000Z, 18500111T0000Z, 18500401T0000Z, …)

5. Running CDDS

The CDDS work flow is driven by the json request file - which holds information about the MIP, the experiment, streams to be processed, start and end dates, the source model suite id, and more.

Where possible you should generate the json request file - you will need access to Met Office internal systems to do that. However, it is not difficult to modify an existing request file for your use - you will need knowledge of the MIP and the experiment for this.

Ocean data preparation

NEMO output needs to be pre-processed to remove halos. CMS have developed a Rose suite to automate halo removal - the suite is available here (??). We suggest creating a fully halo-removed data set prior to running CDDS.

There follows an example work flow, illustrative of the CDDS process. The MIP is AerChemMIP and the experiment is piClim-NTCF (note, case sensitive) - (see https://rawgit.com/WCRP-CMIP/CMIP6_CVs/master/src/CMIP6_experiment_id.html for information about individual experiments.)

Create a directory for the experiment (AerchemMIP-piClim-NTCF). Get cdds-work (from where) - this script orchestrates the conversion process.

export CDDS_DIR="/group_workspaces/jasmin4/ncas_cms/glister/AerchemMIP-piClim-NTCF" REQUEST_JSON=aerchemmip-req.json