wiki:Docs/PostProcessingApp

The Post Processing and PP Transfer App for ARCHER2/NEXCS

This page refers to the Rose post processing App used within cycling suites to post-process data (e.g convert to pp) and optionally to transfer the data to JASMIN. This is valid from UM version 10.0 onwards. The app archives and deletes model output files, not only for the UM, but also for NEMO, including MEDUSA if appropriate, and CICE in coupled configurations. The app is designed to run once at the end of each cycle of a model run.

Post-processing for ARCHER2 and the ARCHER2/NEXCS-JASMIN transfer app was accepted onto the Met Office MOCI trunk and is included in release version postproc_2.2 and above.

Upgrading the Post Processing App

General instructions for upgrading the Post Processing App (which includes JASMIN transfer) can be found on the MOSRS MOCI page.

Configuring postproc

Postproc can be configured for ARCHER2 and NEXCS to post-process data and move it to a staging area ready for transfer to JASMIN by the PP Transfer task.

  1. In panel "postproc → Post Processing - common settings" select the appropriate archive_command.
  1. In panel "postproc → Post Processing - common settings → Archer Archiving" set the following:
    • archive_root_path - The directory you wish to stage data to. Usually somewhere under /work/n02/n02/<username> for ARCHER2 or /projects/nexcs-n02/<username> for NEXCS. This will have the archive_name (by default the suite id) appended to it.

The next 5 points (3-7) are for ARCHER2 setup only. Check first that these are not already set for ARCHER2 via the use of an optional configuration override.

  1. In panel "postproc → Post Processing - common settings" delete the paths for the following variables:
    • ncrcat_path
    • ncks_path
  1. In panel "postproc → CICE → Diagnostics → Meaning" set:
    • means_cmd to be ncra -64bit -O
  1. In panel "postproc → NEMO" the path to the rebuild_nemo executable should be:
    • exec_rebuild to be $UMDIR/nemo/utils/src/REBUILD_NEMO/BLD/bin/rebuild_nemo.exe
  1. In panel "postproc → NEMO → Diagnostics → Meaning" set:
    • means_cmd to be $UMDIR/nemo/utils/bin/mean_nemo_wrapper.py
    • ncatted_cmd to be ncatted
  1. In panel "postproc → Atmosphere" set:
    • um_utils to be $UMDIR/vn10.7/cce/utilities, replacing vn10.7 with the appropriate UM version as necessary.
      Some suites will have this set to $UMDIR/$VN/$PLATFORM/utilities in which case there is nothing to change here.

Configuring Transfer App

Optionally, it is possible to initiate a transfer of the files from ARCHER2 or NEXCS onto JASMIN - this is called the transfer app and is a variation of the postproc app.

If you wish to transfer your archived data to JASMIN, you will need to do some further setup.

In panel "postproc → JASMIN Transfer" set:

  • transfer_dir - Enter the directory on JASMIN to transfer to. This should be one of the group workspaces.
  • transfer_type - This should be set to "Push" for both ARCHER2 and NEXCS.
  • remote_host - This is the host to which you are pushing data to (e.g. hpxfer1.jasmin.ac.uk or xfer[1-2].jasmin.ac.uk)
  • verify_chksums - Select "True" if you wish the transfer app to verify integrity of the transferred files by verifying their checksums (This is the recommended setting).
  • gridftp - Set this option to "False" (Gridftp is currently not available on ARCHER2 or NEXCS)

The transfer runs either on the ARCHER2 login node or in the shared queue on NEXCS.

Note: There are several different data transfer nodes at JASMIN; xfer[1-2].jasmin.ac.uk & hpxfer1.jasmin.ac.uk. Full details on these are available on the JASMIN website. In general if you will be transferring a lot of data you should apply for access to the High Performance Data Transfer Node; hpxfer1.jasmin.ac.uk.

ARCHER2 setup instructions

NEXCS setup instructions

Last modified 6 months ago Last modified on 25/04/21 15:25:35