wiki:Projects/INSPECT/ShaCoNEMO-ORCA1-ARCHER

Version 6 (modified by annette, 3 years ago) (diff)

IPSL ORCA1 on ARCHER

Instructions for running the ShaCoNEMO ORCA1 configuration on ARCHER.

Access

You need to register with the NEMO website to extract the code: http://www.nemo-ocean.eu/user/register

The ORCA1 configuration is held in the ShaCoNEMO repository. You need to register for access, and this uses the same username and password as the main NEMO repository.

To create a branch to hold your code changes on the NEMO repository, you will need additional permissions.

References

The main NEMO website gives more general instructions on building & running NEMO with the standard confgurations included in the code: http://www.nemo-ocean.eu/Using-NEMO/User-Guides/Basics/NEMO-Quick-Start-Guide

The ORCA1_LIM3_PISCES configuration is documented on the ShaCoNEMO (Shared Configuration NEMO) wiki: https://forge.ipsl.jussieu.fr/shaconemo/wiki/README_ORCA1_LIM3_PISCES

  • LIM3 is the sea-ice
  • PISCES is the biogeochemical model

NEMO uses an external library to handle model output diagnostics called XIOS, which is documented here: http://forge.ipsl.jussieu.fr/ioserver/wiki

Code versions

Here we have used the following versions, based on the NEMO/ShaCoNEMO pages:

  • XIOS-1.0 r703
  • NEMO_3.6_stable r6969
  • ShaCoNEMO r60
  • ORCA1 input data set v9

Note: Although XIOS-2.0 is now recommended, revision 932 which is suggested for ORCA1 gave a link-time error which was not investigated.

Note: This is based on the latest version of ShaCoNEMO as of October 2016, and the recommended version of NEMO for this. Both are under active development, so you may wish to use newer versions as they become available.

Code extract

First you need to check out the code from the NEMO repository.

  1. On ARCHER, navigate to the directory that you wish to hold the code. If you are making changes to the standard configuration, you may wish to use the home directory as this is backed up.
  1. To checkout the code to a directory called NEMOGCM_3.6_r6969, run the following, replacing mylogin with your NEMO username (you will be prompted for your password):
    svn --username "mylogin" co -r6969 http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE/NEMOGCM NEMOGCM_3.6_r6969
    

This creates a subversion "working copy". Running svn info lists the revision that was checked out, and svn diff lists the changes that have been made.

  1. To setup and extract the ORCA1 configuration:
    cd NEMOGCM_3.6_r6969/CONFIG
    

Then edit cfg.txt to add the line:

ORCA1_LIM3_PISCES OPA_SRC LIM_SRC_3 NST_SRC TOP_SRC
  1. Next checkout the ORCA1 configuration using the same username (mylogin) and password as for the NEMO checkout:
    svn --username "mylogin" co http://forge.ipsl.jussieu.fr/shaconemo/svn/trunk/ORCA1_LIM3_PISCES
    

This creates the directory ORCA1_LIM3_PISCES, which is another "working copy".

Code build

To build NEMO, you need to point to the location of XIOS, and the parallel-netCDF libraries. A version of XIOS-1 can be found here:

/work/n02/n02/annette/nemo/xios-1.0_r703

It is advisable to build NEMO with the same key modules as the library. The above version was built with:

  • cce/8.4.1
  • cray-netcdf-hdf5parallel/4.3.3.1
  • cray-mpich/7.2.6

(To build your own XIOS, see the notes below.)

To build the NEMO code:

  1. Navigate to the directory holding the build definition files:
    cd NEMOGCM_3.6_r6969/ARCH
    

Then create a new ARCHER configuration file in this directory arch-XC30_NCAS_ARCHER.fcm:

wget http://cms.ncas.ac.uk/attachment/wiki/CmsOnly/Annette/NEMO-howto/arch-XC30_NCAS_ARCHER.fcm

Note this uses the compile flags used for the Met Office GO6 runs.

  1. Go to the config directory:
    cd ../CONFIG
    
    And create a batch submission script for the compilation build_nemo.pbs:
    wget http://cms.ncas.ac.uk/attachment/wiki/CmsOnly/Annette/NEMO-howto/build_nemo.pbs
    
    Edit the PBS headers in the file to set your own project code:
    #PBS -A <xxx>
    
    This build script sets the ARCHER modules required to build with the XIOS library described above, plus it removes the pre-processor flag key_xios2 for building with XIOS-1, and adds key_nosignedzero.
  1. Then submit the build script:
    qsub build_nemo.pbs
    
    This creates a new directory under CONFIG/ called O1L3P. This differentiates between changes made to the original configuration. O1L3P will contain the executable plus namelists to be read in by the model.

Model run

Once the build has completed successfully, set up the inputs required for the model to run.

  1. Go to the run directory:
    cd NEMOGCM_3.6_r6969/CONFIG/O1L3P/EXP00
    
    Copy in the file EMPave.dat:
    cp ../../ORCA1_LIM3_PISCES/EXP00/EMPave_old.dat . 
    
  1. In the same directory, copy in the XIOS-1 XML files (by default the configuration is set up for XIOS-2):
    cp -a ../../ORCA1_LIM3_PISCES/EXP00/ForXIOS1/* . 
    
  1. Then edit the namelist files to make any required changes. For a short test run, edit namelist_cfg and set:
    • nn_itend = 160 : Run length of 5 days.
    • nn_stock = 160 : Restart file write at 5 days.
    • jpni = 8, jpnj = 12, jpnij = 96 : Processor decomposition of 8x12.
  1. Assuming you have been working under your home directory so far, the run directory needs to be copied over to the work filesystem (/work/n02/n02/<username>/), as the ARCHER batch nodes cannot see the home filesystem.

Create a directory to hold your NEMO run, for example:

mkdir -p /work/n02/n02/<username>/nemo/NEMOGCM_3.6_r6969_O1L3P

Then copy over the run directory, evaluating any sym-linked files:

cd NEMOGCM_3.6_r6969/CONFIG/O1L3P
rsync -arL EXP00 /work/n02/n02/<username>/nemo/NEMOGCM_3.6_r6969

If you are on the work filesystem already there is not need to create a new run directory.

  1. Download the ORCA1 input data set, and untar. Again, this needs to go on your work space. To avoid having multiple copies of the data, it is recommended to store this outside of your NEMO run directory.

For example:

cd /work/n02/n02/<username>/nemo
wget http://prodn.idris.fr/thredds/catalog/ipsl_public/romr006/SHACONEMO/eORCA1/catalog.html?dataset=DatasetScanipsl_public/romr006/SHACONEMO/eORCA1/INPUTS_ORCA1_LIM3_PISCES_V9.tar
mkdir INPUTS_ORCA1_LIM3_PISCES_V9
tar -xvf INPUTS_ORCA1_LIM3_PISCES_V9.tar -C INPUTS_ORCA1_LIM3_PISCES_V9

Then create sym-links to the data in your NEMO run directory:

cd /work/n02/n02/<username>/nemo/NEMOGCM_3.6_r6969_O1L3P/EXP00
ln -s /work/n02/n02/<username>/nemo/INPUTS_ORCA1_LIM3_PISCES_V9/* . 
  1. Create a parallel batch submission file to submit the run,

Working practices

Use of home Branches Making changes in ORCA1_LIM3 …

Building XIOS

http://forge.ipsl.jussieu.fr/ioserver/wiki/documentation

Edit the build file to point to your version. If using xios-2 … don't remove flag or copy in files.

Attachments (3)

Download all attachments as: .zip