Changes between Version 14 and Version 15 of Docs/PostProcessingApp


Ignore:
Timestamp:
15/01/21 15:44:45 (3 months ago)
Author:
ros
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Docs/PostProcessingApp

    v14 v15  
    1 = The Post Processing App for ARCHER/NEXCS = 
     1= The Post Processing App for ARCHER2/NEXCS = 
    22 
    3 This page refers to the Rose post processing App used within cycling suites to archive data (e.g to the RDF for ARCHER) and optionally to subsequently transfer the data to JASMIN. This is valid from UM version 10.0 onwards. The app archives and deletes model output files, not only for the UM, but also for NEMO, including MEDUSA if appropriate, and CICE in coupled configurations. The app is designed to run once at the end of each cycle of a model run. 
     3This page refers to the Rose post processing App used within cycling suites to post-process data (e.g convert to pp) and optionally to transfer the data to JASMIN. This is valid from UM version 10.0 onwards. The app archives and deletes model output files, not only for the UM, but also for NEMO, including MEDUSA if appropriate, and CICE in coupled configurations. The app is designed to run once at the end of each cycle of a model run. 
    44 
    5 Post-processing for ARCHER and the ARCHER/NEXCS-JASMIN transfer app was accepted onto the Met Office MOCI trunk and is include in release version '''postproc_2.2''' and above. 
     5Post-processing for ARCHER2 and the ARCHER2/NEXCS-JASMIN transfer app was accepted onto the Met Office MOCI trunk and is include in release version '''postproc_2.2''' and above. 
    66 
    77== Upgrading the Post Processing App == 
     
    1111== Configuring postproc == 
    1212 
    13 Postproc can be configured for Archer to archive data to the RDF (Research Data Facility) or for NEXCS to a temporary location on the `/projects` disk before optionally transferring to JASMIN. 
     13Postproc can be configured for ARCHER2 and NEXCS to post-process data and move it to a staging area ready for transfer to JASMIN by the PP Transfer task. 
    1414 
    15151. In panel ''"postproc -> Post Processing - common settings"'' select the appropriate '''archive_command'''. 
    1616 
    17172. In panel ''"postproc -> Post Processing - common settings -> Archer Archiving"'' set the following: 
    18  * '''archive_root_path''' - The directory you wish to archive to. Usually somewhere under `/nerc/n02/n02/<username>` for RDF or `/projects/nexcs-n02/<username>` for NEXCS. This will have the ''archive_name'' (by default the suite id) appended to it. 
     18 * '''archive_root_path''' - The directory you wish to stage data to. Usually somewhere under `/work/n02/n02/<username>` for ARCHER2 or `/projects/nexcs-n02/<username>` for NEXCS. This will have the ''archive_name'' (by default the suite id) appended to it. 
    1919 
    20 The next 5 points (3-7) are for '''Archer''' setup only.  Check first that these are not already set for ARCHER via the use of an optional configuration override. 
     20The next 5 points (3-7) are for '''ARCHER2''' setup only.  Check first that these are not already set for ARCHER2 via the use of an optional configuration override. 
    2121 
    22223. In panel ''"postproc -> Post Processing - common settings"'' delete the paths for the following variables: 
     
    3939== Configuring Transfer App == 
    4040 
    41 Optionally, it is possible to initiate a transfer of the files from the RDF on ARCHER or from NEXCS onto JASMIN - this is called the ''transfer'' app and is a variation of the ''postproc'' app.  
     41Optionally, it is possible to initiate a transfer of the files from ARCHER2 or NEXCS onto JASMIN - this is called the ''transfer'' app and is a variation of the ''postproc'' app.  
    4242 
    4343If you wish to transfer your archived data to JASMIN, you will need to do some further setup. 
     
    4646  * '''transfer_dir''' - Enter the directory on JASMIN to transfer to.  This should be one of the group workspaces. 
    4747 
    48   * '''transfer_type''' - This should be set to "Push" for both Archer and NEXCS. 
     48  * '''transfer_type''' - This should be set to "Push" for both ARCHER2 and NEXCS. 
    4949 
    50   * '''remote_host''' - This is the host to which you are pushing data to (e.g. jasmin-xfer1.ceda.ac.uk or jasmin-xfer2.ceda.ac.uk) 
     50  * '''remote_host''' - This is the host to which you are pushing data to (e.g. hpxfer1.jasmin.ac.uk or xfer[1-2].jasmin.ac.uk) 
    5151 
    52   * '''verify_chksums''' - Select "True" if you wish the transfer app to verify integrity of the transferred files by verifying their checksums. 
     52  * '''verify_chksums''' - Select "True" if you wish the transfer app to verify integrity of the transferred files by verifying their checksums (This is the recommended setting). 
    5353 
    54   * '''gridftp''' - Select this option if you wish to use gridftp rather than rsync to transfer the files. (This option is not available on NEXCS) 
     54  * '''gridftp''' - Set this option to "False" (Gridftp is currently not available on ARCHER2 or NEXCS) 
    5555 
    56 The transfer runs either on the Data Transfer Node (dtn02.rdf.ac.uk) for ARCHER or in the shared queue on NEXCS. 
     56The transfer runs either on the ARCHER2 login node or in the shared queue on NEXCS. 
    5757 
    58 '''Note:''' There are several different data transfer nodes at JASMIN; `jasmin-xfer[1-3].ceda.ac.uk`.  Full details on these are available on the [https://help.jasmin.ac.uk/article/192-transfer-servers JASMIN website].  In general if you will be transferring a lot of data you should apply for access to the [http://www.jasmin.ac.uk/services/high-performance-data-transfer/ High Performance Data Transfer Node]; `jasmin-xfer2.ceda.ac.uk`. 
     58'''Note:''' There are several different data transfer nodes at JASMIN; `xfer[1-2].jasmin.ac.uk` & `hpxfer1.jasmin.ac.uk`.  Full details on these are available on the [https://help.jasmin.ac.uk/article/192-transfer-servers JASMIN website].  In general if you will be transferring a lot of data you should apply for access to the [http://www.jasmin.ac.uk/services/high-performance-data-transfer/ High Performance Data Transfer Node]; `hpxfer1.jasmin.ac.uk`. 
    5959 
    60 == ARCHER setup instructions == 
     60== ARCHER2 setup instructions == 
    6161 
    62 * PostProcessingAppArcherSetup 
     62* PostProcessingAppArcher2Setup 
    6363 
    6464== NEXCS setup instructions ==