Changes between Version 13 and Version 14 of Docs/PostProcessingApp


Ignore:
Timestamp:
29/11/18 15:52:24 (11 months ago)
Author:
ros
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Docs/PostProcessingApp

    v13 v14  
    33This page refers to the Rose post processing App used within cycling suites to archive data (e.g to the RDF for ARCHER) and optionally to subsequently transfer the data to JASMIN. This is valid from UM version 10.0 onwards. The app archives and deletes model output files, not only for the UM, but also for NEMO, including MEDUSA if appropriate, and CICE in coupled configurations. The app is designed to run once at the end of each cycle of a model run. 
    44 
     5Post-processing for ARCHER and the ARCHER/NEXCS-JASMIN transfer app was accepted onto the Met Office MOCI trunk and is include in release version '''postproc_2.2''' and above. 
     6 
    57== Upgrading the Post Processing App == 
    68 
    7 === Moving from UM Repository to MOCI - Version postproc_1.0 === 
     9General instructions for upgrading the Post Processing App (which includes JASMIN transfer) can be found on the [https://code.metoffice.gov.uk/trac/moci/wiki/app_postproc "MOSRS MOCI"] page. 
    810 
    9 ''Note: This only applies if you are using a UM version prior to version 10.3'' 
    10  
    11 The post processing app originally resided in the UM repository, but was removed prior to the release of version 10.3. In January 2016 postproc_1.0 was released on the MOSRS MOCI repository, and users should upgrade all suites currently using the UM repository to use the new version. A [https://code.metoffice.gov.uk/trac/moci/wiki/app_postproc_umtomoci short guide is available on the MOSRS] for users needing to make this upgrade.  
    12  
    13 === Upgrading to the ARCHER/NEXCS Post Processing Release Version === 
    14  
    15 The post processing for ARCHER has now been accepted onto the Met Office MOCI trunk and was included in release version postproc_2.2. 
    16  
    17 Please ensure that the Rose Editor is closed before attempting to upgrade.  These instructions assume you are upgrading using centrally held meta-data. 
    18  
    19 1. Upgrade the fcm_make_pp app: 
    20  
    21   * cd to `app/fcm_make_pp` 
    22  
    23   * Make sure the app is using centrally held metadata \\ (In `rose-app.conf` the `meta=` should point to a path like `archive_and_meaning/fcm_make/postproc_x.y`.  Where `postproc_x.y` is the current version of the postproc app.) 
    24  
    25   * Run `rose app-upgrade -a postproc_2.2` 
    26  
    27 2. Upgrade the postproc app: 
    28  
    29   * cd to `app/postproc`  
    30  
    31   * Make sure the app is using centrally held metadata \\ (In `rose-app.conf` the `meta=` should point to a path like `archive_and_meaning/postproc/postproc_x.y`.  Where `postproc_x.y` is the current version of the postproc app.) 
    32  
    33   * Run the command: `rose app-upgrade -a postproc_2.2` 
    34  
    35  If it fails to find the upgrade tag postproc_2.2 then run the command `rose app-upgrade -a` to list all available tags to check what is listed. You should see something like: 
    36 {{{ 
    37 $ rose app-upgrade -a 
    38 = postproc_2.1 
    39   pp21_t194 
    40   pp21_t236 
    41   pp21_t221 
    42   pp21_t238 
    43   pp21_t265 
    44   pp21_t122 
    45 * postproc_2.2 
    46 }}} 
    47  
    48 === Configuring fcm_make_pp === 
    49  
    50 In Rose edit navigate to panel ''"fcm_make_pp -> Configuration"'' and make the following changes: 
    51  
    52 1. Set '''config_base''' to: `fcm:moci.xm-tr` 
    53  
    54 2. Set '''config_rev''' to: `@postproc_2.2` 
    55  
    56 3. Set '''pp_rev''' to: `postproc_2.2` 
    57  
    58 4. Remove old branches that refer to old postproc versions. 
    59  
    60 === Configuring postproc === 
     11== Configuring postproc == 
    6112 
    6213Postproc can be configured for Archer to archive data to the RDF (Research Data Facility) or for NEXCS to a temporary location on the `/projects` disk before optionally transferring to JASMIN. 
     
    6718 * '''archive_root_path''' - The directory you wish to archive to. Usually somewhere under `/nerc/n02/n02/<username>` for RDF or `/projects/nexcs-n02/<username>` for NEXCS. This will have the ''archive_name'' (by default the suite id) appended to it. 
    6819 
    69 The next 5 points (3-7) are for '''Archer''' setup only. 
     20The next 5 points (3-7) are for '''Archer''' setup only.  Check first that these are not already set for ARCHER via the use of an optional configuration override. 
    7021 
    71223. In panel ''"postproc -> Post Processing - common settings"'' delete the paths for the following variables: 
     
    7627 * '''means_cmd''' to be `ncra -64bit -O` 
    7728 
    78 5. In panel ''"postproc -> NEMO"'' set: 
     295. In panel ''"postproc -> NEMO"'' the path to the `rebuild_nemo` executable should be: 
    7930 * '''exec_rebuild''' to be `$UMDIR/nemo/utils/src/REBUILD_NEMO/BLD/bin/rebuild_nemo.exe` 
    8031 
     
    8435 
    85367. In panel ''"postproc -> Atmosphere"'' set:  
    86  * '''um_utils''' to be `$UMDIR/vn10.7/cce/utilities`, replacing `vn10.7` with the appropriate UM version as necessary.  
     37 * '''um_utils''' to be `$UMDIR/vn10.7/cce/utilities`, replacing `vn10.7` with the appropriate UM version as necessary. \\ Some suites will have this set to `$UMDIR/$VN/$PLATFORM/utilities` in which case there is nothing to change here. 
    8738 
    88 == Transfer of archived files to JASMIN == 
     39== Configuring Transfer App == 
    8940 
    9041Optionally, it is possible to initiate a transfer of the files from the RDF on ARCHER or from NEXCS onto JASMIN - this is called the ''transfer'' app and is a variation of the ''postproc'' app.  
     
    9445In panel ''"postproc -> JASMIN Transfer"'' set: 
    9546  * '''transfer_dir''' - Enter the directory on JASMIN to transfer to.  This should be one of the group workspaces. 
     47 
    9648  * '''transfer_type''' - This should be set to "Push" for both Archer and NEXCS. 
     49 
    9750  * '''remote_host''' - This is the host to which you are pushing data to (e.g. jasmin-xfer1.ceda.ac.uk or jasmin-xfer2.ceda.ac.uk) 
     51 
    9852  * '''verify_chksums''' - Select "True" if you wish the transfer app to verify integrity of the transferred files by verifying their checksums. 
     53 
    9954  * '''gridftp''' - Select this option if you wish to use gridftp rather than rsync to transfer the files. (This option is not available on NEXCS) 
    10055 
    10156The transfer runs either on the Data Transfer Node (dtn02.rdf.ac.uk) for ARCHER or in the shared queue on NEXCS. 
    10257 
    103 [To Do: Add a note about the 2 JASMIN transfer nodes.] 
     58'''Note:''' There are several different data transfer nodes at JASMIN; `jasmin-xfer[1-3].ceda.ac.uk`.  Full details on these are available on the [https://help.jasmin.ac.uk/article/192-transfer-servers JASMIN website].  In general if you will be transferring a lot of data you should apply for access to the [http://www.jasmin.ac.uk/services/high-performance-data-transfer/ High Performance Data Transfer Node]; `jasmin-xfer2.ceda.ac.uk`. 
    10459 
    10560== ARCHER setup instructions ==