Changes between Version 32 and Version 33 of CDDS


Ignore:
Timestamp:
29/11/19 13:39:50 (12 months ago)
Author:
grenville
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • CDDS

    v32 v33  
    1414== 3. Where to run == 
    1515The CDDS work flow comprises several phases 
    16 A. preparation 
    17 B. data reformatting 
    18 C. quality check 
     16 preparation 
    1917 
    20 Phase A runs interactively, is light on computational resource and can be run on jasmin-cylc. Phase B is computationally intensive, runs on LOTUS, and monitored from jasmin-cylc. Phase C runs interactively but '''should not''' be run on jasmin-cylc - one of the jasmin-sci machines will be appropriate. 
     18 data reformatting 
     19 
     20 quality check 
     21 
     22The preparation phase runs interactively, is light on computational resource and can be run on jasmin-cylc. Data reformatting is computationally intensive, runs on LOTUS, and monitored from jasmin-cylc. The quality runs interactively but '''should not''' be run on jasmin-cylc - one of the jasmin-sci machines will be appropriate. 
    2123 
    2224Note: CDDS sources its own bespoke environment. To avoid potential conflicts, your environment should be as simple as possible - you may need to alter it to allow CDDS to run 
     
    3032The CDDS work flow is driven by the ''json request file'' - which holds information about the MIP, the experiment, streams to be processed, start and end dates, the source model suite id, and more.  
    3133 
    32 Where possible you should generate the json request file - you will need access to Met Office internal systems to do that. However, it might not difficult to modify an existing request file for your use - you will need knowledge of the MIP and the experiment for this. 
     34Where possible you should generate the json request file - you will need access to Met Office internal systems to do that. However, it might not difficult to modify an existing request file for your use - you will need knowledge of the MIP and the experiment for this. An example file is shown below. 
    3335 
    3436=== Ocean data preparation === 
     
    4951}}} 
    5052 
    51  
    52533. Copy  the conversion process orchestration script {{{cdds_workflow_for_user.sh}}} from {{{central-location}}} to this directory 
    5354{{{ 
    54 cp /home/users/glister/CDDS/cdds_workflow_for_user.sh cdds_workflow_for_user.sh 
     55cp /home/users/glister/jasmin-cdds/operational_scripts/cdds_workflow_for_user_v121.sh cdds_workflow_for_user.sh 
    5556}}}  
    56 '''Note: `/home/users/glister/CDDS` is for testing only -- to be moved centrally''' 
     57'''Note: `/home/users/glister/jasmin-cdds/operational_scripts` is for testing only -- to be moved centrally''' 
    5758 
    58594. Edit {{{cdds_workflow_for_user.sh}}} to set the environment variables {{{CDDS_DIR}}}, {{{REQUEST_JSON}}} and {{{FILEPATHSTYPE}}} 
     
    6465Note: call the REQUEST_JSON file something memorable - there may be several in your workflow. 
    6566 
    66 5. Get a copy of {{{CMIP6.cfg}}} from {{{central-location}}} and lodge it in {{{CDDS_DIR}}} under the CMIP6 directory as follows 
    67 {{{ 
    68 mkdir -p CMIP6/v1.0.5/general 
    69 cp /home/users/glister/CDDS/CMIP6.cfg CMIP6/v1.0.5/general/CMIP6.cfg 
    70 }}} 
    71 '''Note: `/home/users/glister/CDDS` is for testing only -- to be moved centrally''' 
    72  
    73 6. Edit your newly copied {{{CMIP6.cfg}}} to set {{{dataroot}}} and {{{procroot}}}. These should be under {{{$CDDS_DIR}}} set in 4. above - in this particular example the edits are  
    74 {{{ 
    75 dataroot = /group_workspaces/jasmin4/ncas_cms/glister/AerChemMIP-piClim-NTCF/cdds_data 
    76 procroot = /group_workspaces/jasmin4/ncas_cms/glister/AerChemMIP-piClim-NTCF/cdds_proc 
    77 }}}  
    78 This just says point {{{dataroot}}} and {{{procroot}}} to {{{cdds_data}}} and {{{cdds_proc}}} (respectively), which will be created later in {{{CDDS_DIR}}}. Note: you '''can not''' use the environment variable `$CDDS_DIR` in {{{CMIP6.cfg}}}. 
    79  
    80 7. Create the json request file, naming the same as {{{$REQUEST_JSON}}} set in 4. -- here's the one used in our example. Much of the information listed here is taken directly from the rose-suite.info file for the UM model suite that generated the data (u-bh543 for our example) 
     675. Create the json request file, naming the same as {{{$REQUEST_JSON}}} set in 4. -- here's the one used in our example. Much of the information listed here is taken directly from the rose-suite.info file for the UM model suite that generated the data (u-bh543 for our example) 
    8168{{{ 
    8269{ 
     
    115102}}} 
    116103 
    117 8. Begin the CDDS process: 
     1046. Begin the CDDS process: 
    118105{{{ 
    119106source cdds_workflow_for_user.sh 
     
    135122 
    136123 
    137  
    138 9. Tell CDDS where the input data resides by specifying its location (through soft links in this case) in the cdds_data directory. In our example, the raw data is in the aerchemmip group workspace 
     1247. Tell CDDS where the input data resides by specifying its location (through soft links in this case) in the cdds_data directory. In our example, the raw data is in the aerchemmip group workspace 
    139125{{{ 
    140126cd $CDDS_DIR/cdds_data/CMIP6/AerChemMIP/UKESM1-0-LL/piClim-NTCF/r1i1p1f2/round-1-monthly/input 
     
    142128}}} 
    143129 
    144 10. Enable the data conversion process - simply un-comment the final {{{cdds_convert}}} command in {{{cdds_workflow_for_user.sh}}} and {{{source cdds_workflow_for_user.sh}}} again. A small amount of work will be repeated, but much of the structure already configured (in particular {{{cdds_data}}}) will persist. [ Note: this should be handled more elegantly through arguments to {{{cdds_workflow_for_user.sh}}} ] 
     1308. Enable the data conversion process - simply un-comment the final {{{cdds_convert}}} command in {{{cdds_workflow_for_user.sh}}}. A small amount of work will be repeated, but much of the structure already configured (in particular {{{cdds_data}}}) will persist. [ Note: this should be handled more elegantly through arguments to {{{cdds_workflow_for_user.sh}}} ] 
    145131 
    146132Several Rose suites will be created - monitor progress on jasmin-cylc with {{{cylc gscan}}}. Suite logging is in cylc-run as usual. 
     
    165151Set up the CDDS environment -  
    166152{{{ 
    167 source /<central location>/setup_cdds_env.sh 
     153source /gws/smf/j04/cmip6_prep/cdds-env/setup_cdds_env.sh 
    168154}}} 
    169155Run the quality check utility