Changes between Version 17 and Version 18 of CDDS


Ignore:
Timestamp:
01/10/19 15:50:06 (6 months ago)
Author:
grenville
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • CDDS

    v17 v18  
    3434=== Ocean data preparation === 
    3535 
    36 NEMO output needs to be pre-processed to remove halos. CMS have developed a Rose suite to automate halo removal - the suite is available here (??). We suggest creating a fully halo-removed data set prior to running CDDS. 
     36Several NEMO data sets need to be pre-processed to remove halos prior to CDDS processing. CMS have developed a Rose suite (u-bn255) to automate halo removal - the suite is available from the MOSRS suite repository . We suggest creating a fully halo-removed data set prior to running CDDS. 
    3737 
    3838=== Example conversion process === 
    3939 
    4040There follows an example work flow, illustrative of the CDDS process. 
    41 The MIP is AerChemMIP and the experiment is piClim-NTCF (note, case sensitive) - (see https://rawgit.com/WCRP-CMIP/CMIP6_CVs/master/src/CMIP6_experiment_id.html for information about individual experiments.) 
     41The example is specific to AerChemMIP for experiment piClim-NTCF - see https://rawgit.com/WCRP-CMIP/CMIP6_CVs/master/src/CMIP6_experiment_id.html for information about individual experiments. 
    4242 
    43 1. Create a top-level directory for the experiment (I chose AerchemMIP-piClim-NTCF in this case). 
    44 2. Copy  the conversion process orchestration script {{{cdds_workflow_for_user.sh}}} (from where?) to this directory 
     43 
     441. Create a top-level directory for the experiment - this will later become synonymous with $CDDS_DIR (I chose to name it `AerchemMIP-piClim-NTCF` in this case). 
     45 
     462. Change directory to `AerchemMIP-piClim-NTCF` - all work should take place here 
    4547{{{ 
    4648cd AerchemMIP-piClim-NTC 
    47 cp cdds_workflow_for_user.sh AerchemMIP-piClim-NTC/cdds_workflow_for_user.sh 
     49}}} 
     50 
     51 
     523. Copy  the conversion process orchestration script {{{cdds_workflow_for_user.sh}}} to this directory 
     53{{{ 
     54cp /<central-location>/cdds_workflow_for_user.sh cdds_workflow_for_user.sh 
    4855}}}  
    4956 
    5057 
    51 3. Edit {{{cdds_workflow_for_user.sh}}} to set {{{CDDS_DIR}}} and {{{REQUEST_JSON}}} 
     584. Edit {{{cdds_workflow_for_user.sh}}} to set the environment variables {{{CDDS_DIR}}} and {{{REQUEST_JSON}}} 
    5259{{{ 
    5360export CDDS_DIR="<full-path>/AerchemMIP-piClim-NTCF" 
    54 REQUEST_JSON=aerchemmip-req.json 
     61REQUEST_JSON=AerChemMIP-piClim-NTCF-req.json 
    5562}}} 
     63Note: call the json file something memorable - there may be several in your workflow. 
    5664 
    57 4. Get a copy of {{{CMIP6.cfg}}} from {{{somewhere central}}} and lodge it in {{{CDDS_DIR}}} under a new directory as follows 
     655. Get a copy of {{{CMIP6.cfg}}} from {{{central-location}}} and lodge it in {{{CDDS_DIR}}} under the CMIP6 directory as follows 
    5866{{{ 
    5967mkdir -p CMIP6/v1.0.5/general 
     
    6169}}} 
    6270 
    63 5. Edit your newly copied {{{CMIP6.cfg}}} to set {{{dataroot}}} and {{{procroot}}} - in this case the edits are 
     716. Edit your newly copied {{{CMIP6.cfg}}} to set {{{dataroot}}} and {{{procroot}}} - in this particular example the edits are 
    6472{{{ 
    6573dataroot = /group_workspaces/jasmin4/ncas_cms/glister/AerchemMIP-piClim-NTCF/cdds_data 
     
    6876This just says point {{{dataroot}}} and {{{procroot}}} to {{{cdds_data}}} and {{{cdds_proc}}} (respectively), which will be created later in {{{CDDS_DIR}}}. 
    6977 
    70 6. Create the json request file ---more guidance? 
    71 7. Begin the CDDS process: 
     787. Create the json request file ---more guidance? 
     79 
     808. Begin the CDDS process: 
    7281{{{ 
    7382source cdds_workflow_for_user.sh 
    7483}}} 
    7584 
    76 Several directories will be created (their structure will be based on information in the json request file.) 
     85Several directories will be created (their structure will be based on information in the json request file.) It is worth familiarizing yourself with the data structure and its relation to entries in the json request file. 
    7786 
    78 A variable list will be generated in {{{cdds-proc}}} 
     87`cdds-data` will hold spaces for input and output data 
     88{{{ 
     89ls /group_workspaces/jasmin4/ncas_cms/glister/AerchemMIP-piClim-NTCF/cdds_data/CMIP6/AerChemMIP/UKESM1-0-LL/piClim-NTCF/r1i1p1f2/round-1-monthly 
     90input/  output/ 
     91}}} 
    7992 
    80 A set of configuration files will be generated. 
     93`cdds_proc` will contain various configuration files and logging output 
     94{{{ 
     95ls /group_workspaces/jasmin4/ncas_cms/glister/AerchemMIP-piClim-NTCF/cdds_proc/CMIP6/AerChemMIP/UKESM1-0-LL_piClim-NTCF_r1i1p1f2/round-1-monthly 
     96archive/  configure/  convert/  extract/  prepare/  qualitycheck/ 
     97}}} 
    8198 
    82 8. Tell CDDS where the input data resides by specifying its location in the cdds_data directory. In this case, the raw data is in the aerchemmip group workspace 
     99Take a look in `convert` 
     100{{{ 
     101ls convert 
     102log/  u-ak283_JSON/ 
     103}}} 
     104`u-ak283-JSON` is the Rose suite that will run to do the data conversions. 
     105 
     106 
     1079. Tell CDDS where the input data resides by specifying its location (through soft links in this case) in the cdds_data directory. In our example, the raw data is in the aerchemmip group workspace 
    83108{{{ 
    84109cd /group_workspaces/jasmin4/ncas_cms/glister/AerchemMIP-piClim-NTCF/cdds_data/CMIP6/AerChemMIP/UKESM1-0-LL/piClim-NTCF/r1i1p1f2/round-1-monthly/input 
     
    86111}}} 
    87112 
    88 9. Enable the data conversion process - simply un-comment the final {{{cdds_convert}}} command in {{{cdds_workflow_for_user.sh}}} and {{{source cdds_workflow_for_user.sh}}} again. A small amount of work will be repeated, but much of the structure already configured (in particular {{{cdds_data}}}) will persist. 
     11310. Enable the data conversion process - simply un-comment the final {{{cdds_convert}}} command in {{{cdds_workflow_for_user.sh}}} and {{{source cdds_workflow_for_user.sh}}} again. A small amount of work will be repeated, but much of the structure already configured (in particular {{{cdds_data}}}) will persist. [ Note: this should be handled more elegantly through arguments to {{{cdds_workflow_for_user.sh}}} ] 
    89114 
    90 Several Rose suites will be created - monitor progress on jasmin-cylc with {{{cylc-gscan}}}. Site logging is in cylc-run as usual. 
     115Several Rose suites will be created - monitor progress on jasmin-cylc with {{{cylc-gscan}}}. Suite logging is in cylc-run as usual. 
    91116 
    92 10. Converted data is in cdds_data 
     11711. Converted data will be written to the `output` directory in cdds_data 
    93118{{{ 
    94 /group_workspaces/jasmin4/ncas_cms/glister/AerchemMIP-piClim-NTCF/cdds_data/CMIP6/AerChemMIP/UKESM1-0-LL/piClim-NTCF/r1i1p1f2/round-1-monthly/output 
     119ls /group_workspaces/jasmin4/ncas_cms/glister/AerchemMIP-piClim-NTCF/cdds_data/CMIP6/AerChemMIP/UKESM1-0-LL/piClim-NTCF/r1i1p1f2/round-1-monthly/output 
     120ap4/  ap4_concat/  ap4_mip_convert/  ap5/  ap5_concat/  ap5_mip_convert/ 
    95121}}} 
    96122