wiki:Docs/PostProcessingApp

Version 6 (modified by ros, 2 years ago) (diff)

The Post Processing App for ARCHER/NEXCS

This page refers to the Rose post processing App used within cycling suites to archive data (e.g to the RDF for ARCHER) and optionally to subsequently transfer the data to JASMIN. This is valid from UM version 10.0 onwards. The app archives and deletes model output files, not only for the UM, but also for NEMO, including MEDUSA if appropriate, and CICE in coupled configurations. The app is designed to run once at the end of each cycle of a model run.

Upgrading the Post Processing App

Moving from UM Repository to MOCI - Version postproc_1.0

Note: This only applies if you are using a UM version prior to version 10.3

The post processing app originally resided in the UM repository, but was removed prior to the release of version 10.3. In January 2016 postproc_1.0 was released on the MOSRS MOCI repository, and users should upgrade all suites currently using the UM repository to use the new version. A short guide is available on the MOSRS for users needing to make this upgrade.

Upgrading to the ARCHER/NEXCS Post Processing Branch

The post processing for ARCHER is still in development but an initial stable version (r1859) has been released and will hopefully be incorporated into the Met Office MOCI trunk very soon. It is based on the Met Office's most recent post processing version (postproc_2.0).

Please ensure that the Rose Editor is closed before attempting to upgrade.

  1. Upgrade the fcm_make_pp app:
  • cd to app/fcm_make_pp
  • Run rose app-upgrade -a this will give a list of available upgrade/downgrade versions. The version marked with a "=" is the current version. If it is not pp20_t109 then you will need to upgrade it by running rose app-upgrade -a pp20_t109.
  1. Upgrade the postproc app:
  • Modify the first line of app/postproc/rose-app.conf to change meta=…. setting. This will be different dependending on the machine you are working on.
    On PUMA:
    meta=/home/ros/meta/postproc/2.0/r2142/rose-meta/archive_and_meaning/postproc/postproc_2.0
    
    On XCS-C (NEXCS):
    meta=/home/d04/rhatcher/meta/postproc/2.0/r2142/rose-meta/archive_and_meaning/postproc/postproc_2.0
    
  • Close rose-app.conf file
  • cd to app/postproc
  • Run the command: rose app-upgrade -a pp20_t122
    If it fails to find the upgrade tag pp20_t122 then run the command rose app-upgrade -a to list all available tags to check what is listed. You should see something like:
    $ rose app-upgrade -a
    = postproc_2.0
    * pp20_t122
    
  • Open the suite with rose edit.

Following this upgrade there will be a warning (either on opening the suite or viewing the postproc app) that the appropriate metadata version could not be found. You will then need to click on postproc and change meta to pick up the new metadata changes. You should simply need to replace the trailing pp20_t122 with HEAD and click "Apply".

For reference the path should be:
Archer: /home/ros/meta/postproc/2.0/r2142/rose-meta/archive_and_meaning/postproc/HEAD
NEXCS: /home/d04/rhatcher/meta/postproc/2.0/r2142/rose-meta/archive_and_meaning/postproc/HEAD

Configuring fcm_make_pp

In Rose edit navigate to panel "fcm_make_pp → Configuration" and make the following changes:

  1. Set config_base to: fcm:moci.xm-br/dev/rosalynhatcher/r1901_archer_postproc
  1. Set config_rev to: @2142
  1. To pp_sources add the branch: fcm:moci.xm-br/dev/rosalynhatcher/r1901_archer_postproc@2142
  1. Ensure the following branch is also included: branches/dev/ericaneininger/postproc_2.0_retire_CYLC_SUITE_REG_NAME

Configuring postproc

Postproc can be configured for Archer to archive data to the RDF (Research Data Facility) or for NEXCS to a temporary location on the /projects disk before optionally transferring to JASMIN.

  1. In panel "postproc → Post Processing - common settings" select the appropriate archive_command.
  1. In panel "postproc → Archer Archiving" set the following:
    • archive_dir - The directory you wish to archive to. Usually somewhere under /nerc/n02/n02/<username> for RDF or /projects/nexcs-n02/<username> for NEXCS. This will have the archive_name (by default the suite id) appended to it.

The next 3 points (3-5) are for Archer setup only.

  1. In panel "postproc → CICE → Meaning" set:
    • means_cmd to be ncra -64bit -O
  1. In panel "postproc → NEMOC → Meaning" set:
    • means_cmd to be $UMDIR/nemo/utils/bin/mean_nemo_wrapper.py
    • ncatted_cmd to be ncatted
  1. In panel "postproc → Atmosphere" set:
    • um_utils to be $UMDIR/vn10.6/cce/utilities, replacing vn10.6 with the appropriate UM version as necessary.

Transfer of archived files to JASMIN

Optionally, it is possible to initiate a transfer of the files from the RDF on ARCHER or from NEXCS onto JASMIN - this is called the transfer app and is a variation of the postproc app.

If you wish to transfer your archived data to JASMIN, you will need to do some further setup.

In panel "postproc → JASMIN Transfer" set:

  • transfer_dir - Enter the directory on JASMIN to transfer to. This should be one of the group workspaces.
  • transfer_type - This should be set to "Push" for both Archer and NEXCS.
  • remote_host - This is the host to which you are pushing data to (e.g. jasmin-xfer1.ceda.ac.uk or jasmin-xfer2.ceda.ac.uk)
  • verify_chksums - Select "True" if you wish the transfer app to verify integrity of the transferred files by verifying their checksums.
  • gridftp - Select this option if you wish to use gridftp rather than rsync to transfer the files. (This option is not available on NEXCS)

The transfer runs either on the Data Transfer Node (dtn02.rdf.ac.uk) for ARCHER or in the shared queue on NEXCS.

[To Do: Add a note about the 2 JASMIN transfer nodes.]

ARCHER setup instructions

NEXCS setup instructions