Version 46 (modified by cthomas, 6 years ago) (diff)

Notes on NEMOVAR


Operation of the NEMOVAR program is controlled by the Rose suite labelled puma-aa164, which is a copy of puma-aa145.

Suite setup


The structure of the suite is the standard one:

  • suite.rc: suite structure and operation
  • basic suite info
  • rose-suite.conf: configuration of variables used in the suite (including paths)
  • meta/rose-meta.conf: information about the variables defined in rose-suite.conf


These are the variables that are defined in rose-suite.conf and used in suite.rc.

  • START_POINT: Start point of run
  • RUN_LENGTH: Length of run
  • CYCLE_LENGTH: Length of cycle
  • WALL_CLOCK_LIMIT: Wall clock limit
  • CALENDAR: Calendar used (default '365day')
  • CICE_COL: Number of CICE columns
  • CICE_MAXBK: Maximum number of blocks per processor for CICE
  • CICE_ROW: Number of CICE rows
  • COMPUTE_HOST: Host to use for compilation and/or model run
  • EXTRACT_HOST: Host to use for code extraction
  • NEMO_IPROC: Number of NEMO processes E-W
  • NEMO_JPROC: Number of NEMO processes N-S
  • RUNID: Prefix run ID for output files
  • BUILD: Build model? (Y/N)
  • RUN: Run model? (Y/N)
  • GROUP: User group

Directories included

The following directories are included in rose-suite.conf:

  • file:app/daily_fluxes
  • file:app/daily_observations
  • file:app/fixed_ancillary
  • file:app/fixed_restarts
  • file:app/nemovar
  • file:app/nemo_cice
  • file:app/fcm_make_nemo

By default each directory points towards a location on SVN. The locations are all subdirectories of the following directory:


Each directory can be checked out locally and modified, in which case the path specified in rose-suite.conf must be changed to point to the local version.

The directories all contain a rose-app.conf file along with other files/subdirectories. Each directory is discussed in more detail below.


This corresponds to the subdirectory operational_atmospheric_forcing. This app copies the daily flux files.

  • app-interface.conf: defines IN_ADDITIONAL_DAYS and OUTDIR_FLUXES
  • rose-app.conf: sets the default command to prepare_fluxes, defines INPUT_DIR
  • bin/prepare_fluxes: Script to copy flux files from INPUT_DIR to OUTDIR_FLUXES and unzip them
  • opt/rose-app-FOAM_v13.conf: defines INPUT_DIR (this is overridden by rose-app.conf)


This corresponds to the subdirectory hindcast_observations. This app copies the daily observation files. Everything in the opt directory controls a different set of observations.

  • app-interface.conf: defines OUTDIR_OBSERVATIONS
  • rose-app.conf: defines INBASEDIR, OUTDIR and EXEC_DIR and sets a large number of namelist parameters
  • bin/NemoQcProg_ExtractAndProcess: Script to copy observation files in place and run the NEMO QC to file convert to 'feedback' format
  • bin/NemoScr_ObsPreProc: Script to run the NEMOQC code on operational observations
  • opt/rose-app-{altimeter,ghrsst_avhrr,ghrsst_metop,profile,seaice1,seaice2,surface}.conf: (re)defines INSUBDIR, INOBSFILE and OUTOBSFILE and sets a namelist parameter


This corresponds to the subdirectory GO5_orca025_ancillary. This app copies ancillary files into the output directory.

  • rose-app.conf: defines a variety of environment variables (pointing to NEMO data directories) and creates softlinks. The environment variables are: NEMO_ANCIL, NEMO_FORCE, NEMO_GRIDS, NEMO_INPUTS, NEMO_EXECS and NEMO_IODEF
  • bin/copy_ancilliary: empty
  • bin/operational_ancilliary: creates a large number of softlinks


This corresponds to the subdirectory initial_restarts. This app copies the initial restarts.

  • rose-app.conf: defines RESTART_DIR
  • bin/copy_restarts: creates a link from RESTART_DIR to OUTDIR_RESTARTS


This corresponds to the subdirectory nemovar. This app configures NEMOVAR.

  • rose-app.conf: defines a large number of namelist parameters
  • bin/ two helper functions
  • bin/ initialise NEMOVAR
  • bin/ run NEMOVAR
  • bin/ calculate SST Bias
  • file/iodef.xml: standard NEMO file (controls fields and outputs)
  • file/xmlio_server.def: standard NEMO file
  • opt/rose-app-orca025.conf: defines a large number of namelist parameters
  • opt/rose-app-sstbias-orca025.conf: defines a large number of namelist parameters


This corresponds to the subdirectory GO5_nemo_GSI6_cice. This app configures NEMO-CICE.

  • rose-app.conf: defines a large number of namelist parameters
  • bin/ two helper functions
  • bin/run_nemo_cice: patched version removes handle_observations and processing profb
  • bin/update_nemo_nl: script to update namelists
  • meta/rose-meta.conf: large number of definitions including of namelist variables
  • opt/rose-app-iau.conf: namelist definitions for iau
  • opt/rose-app-obsoper.conf: namelist definitions for obsoper


This corresponds to the subdirectory fcm_make_puma_GO5_nemo_GSI6_cice. This app builds NEMO and related programs.

  • rose-app.conf: empty
  • file/fcm-make.cfg: links to other .cfg files
  • file/fcm-make-GO5.cfg: for compiling NEMO
  • file/fcm-make-GSI5.cfg: for compiling CICE
  • file/pwr6-xlf-opt.cfg: similar to a Makefile


Details of the version of NEMO-CICE used will go here.


Description of the more interesting namelist variables will go here.

Suite operation

This is a rough description of what happens when the suite runs. An example cylc dependency tree can be seen on this page. The suite operates in two stages: the first stage involves compilation and setup and the second stage is where the cycling occurs.

First stage:

  • build nemo
  • run fixed_ancillary (pulls in ancillary information)
  • run fixed_restarts (the background)
  • if all successful, run nemo_cice_obsoper (the observation operator step)

Remaining stages:

  • run daily_fluxes
  • if all successful, run nemo_cice_obsoper (produces innovations & forecast background)
  • DAILY_OBSERVATIONS 'suicides' if nemo_cice_obsoper succeeds
  • if nemo_cice_obsoper succeeds, run nemovar (produces increments)
  • if nemovar succeeds, run nemo_cice_iau at a restart 'P1D' (incremental analysis update - produces analysis restarts)
  • if nemo_cice_iau succeeds, run nemo_cice_obsoper again


When rose suite-run is executed a cylc task is started. This populates the $HOME/cylc-run/puma-aa164 directory with the following:

  • The app directory contains subdirectories which are listed in the original rose-suite.conf: daily_fluxes, daily_observations, fcm_make_nemo, fixed_ancillary, fixed_restarts, nemo_cice, nemovar
  • suite.rc is copied from the suite directory, adding in the definitions from rose-suite.conf This file is then processed by substituting all of the variables to make suite.rc.processed
  • Two softlinks are also created to the $DATADIR/cylc-run/puma-aa164/{work,share} directories
  • Log files appear as always, the latest of which is linked to by log
  • cylc-suite.db, cylc-suite-env,
  • state directory with current states at different times
  • meta/rose-meta.conf is the same as in the suite directory

Input data

Information on the input data can be found here.

Output data

Output is placed in $DATADIR/cylc-run/puma-aa164/work and $DATADIR/cylc-run/puma-aa164/share.

As stated on the Instructions page the data take up at the moment:

  • work: 90 GB per day
  • share: 4 GB initial (fcm_make_nemo and data/ancillary) + 5.5 GB per run day

The contents of each directory are described in the next two sections.


The subdirectories are:

  • cycle: output from each cycle (subdirectories analysis, assim_background, background, fluxes, increments, innovations, observations, updated_altbias)
  • data: ancillary data (various .nc)
  • fcm_make_nemo: output of the build procedure (e.g. .o, .mod, .f90 files)


There are subdirectories for each day run over. Each subdirectory contains the rose-app-run.conf file that was present in the original directory. The additional files produced in the default setup are as follows. NNNN indicates the particular subjob number (from 0000 to 0192 in the default setup) and DDDD indicates a date.

  • daily_fluxes: nothing else in here
  • daily_observations_{altimeter,ghrsst_avhrr,ghrsst_metop,profile,seaice1,seaice2,surface}: various namelist and .nc files
  • fcm_make_nemo: the .cfg files listed above
  • fixed_ancillary: nothing else in here
  • fixed_restarts: nothing else in here
  • nemo_cice_iau: some NEMO input files,,,,,,,,
  • nemo_cice_obsoper: some NEMO input files,,,,,,,,,,,,,,,,
  • nemovar: some NEMO input files,,,,,,,

Example runs

Default setup

The default setup is to run for two days with one day of cycling, starting on 1/1/11. Much more information can be found here:

Note that for this particular date, no profile data are available, so the "daily_observations_profile" app will fail. The NEMO observation operator can still run without this data, however, and the suite should continue to run.

Different start date and run length

From the Instructions page:

To change the start date and run length, edit the START_POINT and RUN_LENGTH variables in the rose-suite.conf file. You will also need to edit the "daily_fluxes", "daily_observations" and "fixed_restarts" apps to point to the locations of the input data.

Adding other files

To do

  • Where is OUTDIR_RESTARTS set?
  • Define ghrsst, etc.
  • Check profile input data
  • Check output in share and work directories