Changes between Initial Version and Version 1 of Docs/PostProcessingAppArcherSetup


Ignore:
Timestamp:
18/09/17 14:26:00 (3 years ago)
Author:
ros
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Docs/PostProcessingAppArcherSetup

    v1 v1  
     1= ARCHER specific setup instructions for data transfer to JASMIN = 
     2 
     3If you wish to transfer files from the RDF to JASMIN using the transfer app then you will need to make the following changes in addition to those detailed on the Post Processing App page. 
     4 
     5== Suite Modifications == 
     6 
     7 1. Edit `rose-suite-conf`: 
     8 
     9 Add '''PPTRANSFER=true''' 
     10 
     11 2. The PPTRANSFER variable will, by default, appear under ''"suite conf -> jinja2"''.  To tell Rose to place it with all the other suite control switches under ''"suite conf -> Build and Run"'' edit `meta/rose-meta.conf` to add in the metadata for the PPTRANSFER variable.  (This step is optional.) 
     12{{{ 
     13[jinja2:suite.rc=PPTRANSFER] 
     14compulsory=true 
     15description=Transfer files archived with PostProc to a remote machine 
     16help= 
     17ns=Build and Run 
     18sort-key=runPostproc1 
     19title=PP Transfer 
     20type=boolean 
     21}}} 
     22 
     23 3. Add the ''pptransfer'' task into the cylc graph. Edit `suite.rc` file: 
     24 
     25  [To Do: Add in/change to cater for suites with the "SITE" setup] 
     26 
     27  * Add the line:  
     28{{{ 
     29{{ 'fcm_make_pptransfer => fcm_make2_pptransfer' + (' => pptransfer' if RUN else '') if PPTRANSFER else '' }} 
     30}}} 
     31 to the cylc graph for the initial cycle, indicated by ![[[ R1 ]]].  For example: (insertion indicated by ''"<= Add line here"'') 
     32{{{ 
     33    [[dependencies]] 
     34        [[[ R1 ]]] 
     35            graph = """ 
     36{{ 'fcm_make_pp => fcm_make2_pp' + (' => postproc' if RUN else '') if POSTPROC else '' }} 
     37{{ 'fcm_make_pptransfer => fcm_make2_pptransfer' + (' => pptransfer' if RUN else '') if PPTRANSFER else '' }}    <== Add line here 
     38{{ 'fcm_make_ocean => fcm_make2_ocean' + (' => recon' if RECON else ' => coupled' if RUN else '') if BUILD_OCEAN else '' }} 
     39{{ 'fcm_make_um => fcm_make2_um' + (' => recon' if RECON else ' => coupled' if RUN else '') if BUILD_UM else '' }} 
     40{{ 'install_ancil => recon ' if RECON else ('install_ancil => coupled' if RUN else '')}} 
     41{{ 'recon' + (' => coupled' if RUN else '') if RECON else '' }} 
     42{{ 'clearout' + (' => coupled' if RUN else '') if CLEAROUT else '' }} 
     43""" 
     44}}} 
     45 
     46  *Add the ''pptransfer'' task into the graph for all subsequent cycles such that it runs after the ''postproc'' task and also waits for the previous ''pptransfer'' task to complete.  As an example for a coupled suite (All added lines indicated with ''"<="''): 
     47{{{ 
     48        [[[ {{FMT}} ]]] 
     49            graph = """ 
     50{% if RUN %} 
     51coupled[-{{FMT}}] => coupled {{ '=> \\' if POSTPROC or HOUSEKEEP else '' }} 
     52  {% if POSTPROC %} 
     53postproc {{ '=> \\' if PPTRANSFER or HOUSEKEEP else '' }}     <= "PPTRANSFER or" added here 
     54  {% endif %} 
     55  {% if PPTRANSFER %}                                         <= 
     56pptransfer {{ '=> \\' if HOUSEKEEP else '' }}                 <= 
     57  {% endif %}                                                 <= 
     58  {% if HOUSEKEEP %} 
     59housekeeping 
     60  {% endif %} 
     61  {% if POSTPROC %} 
     62postproc[-{{FMT}}] => postproc 
     63  {% endif %} 
     64  {% if PPTRANSFER %}                                         <= 
     65pptransfer[-{{FMT}}] => pptransfer                            <= 
     66  {% endif %}                                                 <= 
     67{% endif %} 
     68""" 
     69}}} 
     70 
     71  * In the `[[postproc]]` task: 
     72    * Check that `pre-script` is set to `"module load nco/4.6.8; module load anaconda; export PYTHONPATH=$PYTHONPATH:$UMDIR/lib/python2.7; module list; ulimit -s unlimited"`: 
     73{{{ 
     74    [[postproc]] 
     75        inherit = ... 
     76        pre-script = "module load nco/4.6.8; module load anaconda; export PYTHONPATH=$PYTHONPATH:$UMDIR/lib/python2.7; module list; ulimit -s unlimited" 
     77        ... 
     78}}} 
     79 
     80  4. At the end of the file add the ''pptransfer'' task definitions: 
     81{{{ 
     82    [[PPTRANSFER]] 
     83        [[[remote]]] 
     84            host = dtn02.rdf.ac.uk 
     85        [[[environment]]] 
     86            UMDIR=~um 
     87 
     88    [[PPTRANSFER_BUILD]] 
     89        [[[environment]]] 
     90            ROSE_TASK_APP=fcm_make_pp 
     91 
     92    [[fcm_make_pptransfer]] 
     93        inherit = None, LINUX_UM, PPTRANSFER_BUILD 
     94 
     95    [[fcm_make2_pptransfer]] 
     96        inherit = None, PPTRANSFER, PPTRANSFER_BUILD 
     97    
     98    [[pptransfer]] 
     99        inherit = PPTRANSFER 
     100        pre-script = "module load anaconda" 
     101        [[[environment]]] 
     102            CYCLEPERIOD = $( rose date $CYLC_TASK_CYCLE_POINT $CYLC_TASK_CYCLE_POINT --calendar {{CALENDAR}} --offset2 {{FMT}} -f y,m,d,h,M,s ) 
     103            ROSE_TASK_APP=postproc 
     104            PLATFORM = linux 
     105}}}