Version 82 (modified by cthomas, 4 years ago) (diff)

Notes on NEMO/NEMOVAR cycling suite for assimilation


Operation of the NEMO/NEMOVAR assimilation code is controlled by the Rose suite labelled puma-aa184, which is based on puma-aa164 (itself a copy of puma-aa145). The purpose of this suite is to assimilate data into a high-resolution ocean model using a cycling framework.

The setup of the suite is described in Suite setup and its operation is described in Suite operation. More detail on the NEMO/NEMOVAR operation can be found in the NEMO/NEMOVAR section. Finally, tips for running different configurations are in the Example suite usage section.

Suite setup


The structure of the suite is the standard one:

  • suite.rc: suite structure and operation
  • basic suite info
  • rose-suite.conf: configuration of variables used in the suite (including paths), picked up in suite.rc
  • meta/rose-meta.conf: information about the variables defined in rose-suite.conf

Information on Rose can be found here and a guide to cylc can be found here.

The suite itself is quite simple and acts as a springboard for many other processes to run.


These are the variables that are defined in rose-suite.conf and used in suite.rc.

  • START_POINT: Start point of run
  • RUN_LENGTH: Length of run
  • CYCLE_LENGTH: Length of assimilation cycle
  • WALL_CLOCK_LIMIT: Wall clock limit
  • CALENDAR: Calendar used (default '365day')
  • CICE_COL: Number of CICE columns
  • CICE_MAXBK: Maximum number of blocks per processor for CICE
  • CICE_ROW: Number of CICE rows
  • COMPUTE_HOST: Host to use for compilation and/or model run
  • EXTRACT_HOST: Host to use for code extraction
  • NEMO_IPROC: Number of NEMO processes E-W
  • NEMO_JPROC: Number of NEMO processes N-S
  • RUNID: Prefix run ID for output files
  • BUILD: Build model? (Y/N)
  • RUN: Run model? (Y/N)
  • GROUP: User group

The most variables which change most frequently are START_POINT, RUN_LENGTH, CYCLE_LENGTH and WALL_CLOCK_LIMIT (which needs to be larger for a longer assimilation cycle).

When assimilating large files it may also be necessary to increase the memory limit. This can be done by altering ConsumableMemory in the relevant place in suite.rc.

Apps included

The suite contains several apps which are used to perform auxiliary tasks such as retrieving satellite data for a particular day. The apps are listed in rose-suite.conf:

  • file:app/daily_fluxes
  • file:app/daily_observations
  • file:app/fixed_ancillary
  • file:app/fixed_restarts
  • file:app/nemovar
  • file:app/nemo_cice
  • file:app/fcm_make_nemo

By default each app points towards a location on SVN. The SVN locations are all subdirectories of the following directory:


Each subdirectory can be checked out locally and modified, in which case the path specified in rose-suite.conf must be changed to point to the local version.

The subdirectories all contain a rose-app.conf file along with other files/subdirectories. Each app is discussed in more detail below.

Daily fluxes

The app is called daily_fluxes and corresponds to the subdirectory operational_atmospheric_forcing. This app copies the daily flux files. For day D and cycle length C, flux files from days (D - 1) to (D + C) must be present in the directory INPUT_DIR.

  • app-interface.conf: defines IN_ADDITIONAL_DAYS and OUTDIR_FLUXES
  • rose-app.conf: sets the default command to prepare_fluxes, defines INPUT_DIR
  • bin/prepare_fluxes: Script to copy flux files from INPUT_DIR to OUTDIR_FLUXES and unzip them
  • opt/rose-app-FOAM_v13.conf: defines INPUT_DIR (this is not used)

The files in INPUT_DIR must be listed in prepare_fluxes:

set -A FILESTEMS <filestem1> <filestem2> ...

where each filestem is the stem before the particular date in the filename. For example, if the stem is flux then the full filename for 1/1/11 would be

prepare_fluxes then copies and unzips the relevant flux file.

It is important to ensure the namelist reflects the contents of these files. This part of the namelist is modified in the NEMO-CICE rose-app.conf (see later).

Daily observations

The app is called daily_observations and corresponds to the subdirectory hindcast_observations. This app copies the daily observation files for assimilation into the particular cycle. Everything in the opt directory controls a different set of observations.

  • app-interface.conf: defines OUTDIR_OBSERVATIONS
  • rose-app.conf: defines INBASEDIR, OUTDIR and EXEC_DIR and sets a large number of namelist parameters
  • bin/NemoQcProg_ExtractAndProcess: Script to copy observation files in place and run the NEMO QC to file convert to 'feedback' format
  • bin/NemoScr_ObsPreProc: Script to run the NEMOQC code on operational observations (not used in this case)
  • opt/rose-app-{altimeter,ghrsst_avhrr,ghrsst_metop,profile,seaice1,seaice2,surface}.conf: (re)defines INSUBDIR, INOBSFILE and OUTOBSFILE and sets a namelist parameter

The observations are as follows:

  • Altimeter: satellite altimetry; AVISO and/or "realtime" measurements
  • GHRSST: sea surface temperature (SST) from two satellites (AVHRR, METOP)
  • Profile: profile measurements
  • Sea ice: sea ice
  • Surface: in-situ SST

The location of these files is controlled in the relevant rose-app-*.conf file; both an input directory (INBASEDIR) and file stem (INOBSFILE) are required. Similarly to the fluxes, the filenames must contain the date. Note that profile data may be monthly instead of daily.

The script NemoQcProg_ExtractAndProcess prepares the files by copying them, unzipping them and running NemoQcProg_ExtractAndProcess.exe, which applies some quality control.

Ancillary information

The app is called fixed_ancillary and corresponds to the subdirectory GO5_orca025_ancillary. This app copies ancillary files (usually one-offs that don't change during the suite's operation) into the output directory.

  • rose-app.conf: defines a variety of environment variables (pointing to NEMO data directories) and creates softlinks. The environment variables are: NEMO_ANCIL, NEMO_FORCE, NEMO_GRIDS, NEMO_INPUTS, NEMO_EXECS and NEMO_IODEF
  • bin/copy_ancilliary: empty
  • bin/operational_ancilliary: creates a large number of softlinks

Example ancillary files include bathymetry, x/y positions, rivers, and XML configuration of NEMO.

Initial restarts

The app is called fixed_restarts and corresponds to the subdirectory initial_restarts. This app copies the initial restarts which are used as the background state on the first assimilation. If the cycling starts on data D, the restarts must be present for that date. In further iterations of the cycle, restarts are taken to be the analysis from the previous iteration; see Suite operation below.

  • rose-app.conf: defines RESTART_DIR
  • bin/copy_restarts: creates a link from RESTART_DIR to OUTDIR_RESTARTS, which is set in suite.rc.


The app is called nemovar and corresponds to the subdirectory nemovar. This app configures NEMOVAR.

  • rose-app.conf: defines a large number of namelist parameters
  • bin/ two helper functions
  • bin/ initialise NEMOVAR
  • bin/ run NEMOVAR
  • bin/ calculate SST Bias
  • file/iodef.xml: standard NEMO file (controls fields and outputs)
  • file/xmlio_server.def: standard NEMO file
  • opt/rose-app-orca025.conf: defines a large number of namelist parameters
  • opt/rose-app-sstbias-orca025.conf: defines a large number of namelist parameters

The two scripts and are very important and are discussed in more detail in the NEMO/NEMOVAR section.


The app is called nemo_cice and corresponds to the subdirectory GO5_nemo_GSI6_cice. This app configures NEMO-CICE. There are two stages: obsoper and IAU. A lot more detail can be found in the Suite operation and NEMO/NEMOVAR sections.

  • rose-app.conf: defines a large number of namelist parameters, including how to read in flux files
  • bin/ two helper functions
  • bin/run_nemo_cice: run the assimilation
  • bin/update_nemo_nl: script to update namelists
  • meta/rose-meta.conf: large number of definitions including of namelist variables
  • opt/rose-app-iau.conf: namelist definitions for iau
  • opt/rose-app-obsoper.conf: namelist definitions for obsoper

The run_nemo_cice script is particularly important here and is described more in the later sections.


The app is called fcm_make_nemo and corresponds to the subdirectory fcm_make_puma_GO5_nemo_GSI6_cice. This app builds NEMO and related programs.

  • rose-app.conf: empty
  • file/fcm-make.cfg: links to other .cfg files
  • file/fcm-make-GO5.cfg: for compiling NEMO
  • file/fcm-make-GSI5.cfg: for compiling CICE
  • file/pwr6-xlf-opt.cfg: similar to a Makefile

Suite operation

This is a rough description of what happens when the suite runs. An example cylc dependency tree can be seen on this page. The suite operates in two parts: the first part involves compilation and setup and the second is where the cycling occurs.

First part:

  • build NEMO/CICE
  • run fixed_ancillary (pulls in ancillary information)
  • run fixed_restarts (the background)
  • if all successful, run nemo_cice_obsoper (the observation operator step)

Second part:

  • obtain daily fluxes
  • obtain daily observations
  • if all successful, run nemo_cice_obsoper
  • if nemo_cice_obsoper succeeds, run nemovar
  • if nemovar succeeds, run nemo_cice_iau
  • if nemo_cice_iau succeeds, run nemo_cice_obsoper at the next step in the cycle

The cycling occurs in the second part. This consists of three stages:

Stage 1: form the observation operator (obsoper). Produces the innovations y - H(xb).

Required input Program run Output
Ancillary NEMO-CICE obsoper Innovations
Restarts Forecast background

In the first cycle the restarts are the background; in subsequent cycles the restarts are the analysis produced in the previous cycle.

Stage 2: run NEMOVAR which minimises the cost function J. Produces the resulting increments Δx.

Required input Program run Output
Innovations NEMOVAR Increments
Forecast background

Stage 3: incremental analysis update (IAU); apply the increments to the background to produce the analysis xa.

Required input Program run Output
Ancillary NEMO-CICE obsoper Analysis (restarts)


When rose suite-run is executed a cylc task is started. This populates the $HOME/cylc-run/puma-aa168 directory with the following:

  • The app directory contains subdirectories which are listed in the original rose-suite.conf: daily_fluxes, daily_observations, fcm_make_nemo, fixed_ancillary, fixed_restarts, nemo_cice, nemovar. These have been discussed above in the suite setup section.
  • suite.rc is copied from the suite directory, adding in the definitions from rose-suite.conf This file is then processed by substituting all of the variables to make suite.rc.processed
  • Two softlinks are also created to the $DATADIR/cylc-run/puma-aa164/{work,share} directories
  • Log files appear as always, the latest of which is linked to by log
  • cylc-suite.db, cylc-suite-env, contain some suite information
  • state directory with current states at different times
  • meta/rose-meta.conf is the same as in the suite directory

Input data

Input data consists of various measurements (SST, altimetry, ice), fluxes, ancillary information and the initial restarts. These inputs have been discussed in the suite setup section. Some more information can be found here.

Output data

Output is placed in $DATADIR/cylc-run/puma-aa168/work and $DATADIR/cylc-run/puma-aa168/share. The contents of each directory are described in the next two sections.

Note: the files produced are subject to change…


The subdirectories are:

  • cycle: output from each cycle (subdirectories analysis, assim_background, background, fluxes, increments, innovations, observations, updated_altbias)
  • data: ancillary data (various .nc files)
  • fcm_make_nemo: output of the build procedure (e.g. .o, .mod, .f90 files)


There are subdirectories for each day run over. Each subdirectory contains the rose-app-run.conf file that was present in the original directory. The additional files produced in the default setup are as follows. NNNN indicates the particular subjob number (from 0000 to 0192 in the default setup) and DDDD indicates a date.

  • daily_fluxes: nothing else in here
  • daily_observations_{altimeter,ghrsst_avhrr,ghrsst_metop,profile,seaice1,seaice2,surface}: various namelist and .nc files
  • fcm_make_nemo: the .cfg files listed above
  • fixed_ancillary: nothing else in here
  • fixed_restarts: nothing else in here
  • nemo_cice_iau: some NEMO input files,,,,,,,,
  • nemo_cice_obsoper: some NEMO input files,,,,,,,,,,,,,,,,
  • nemovar: some NEMO input files,,,,,,,


The NEMO/NEMOVAR step is where the assimilation occurs. It has already been outlined in Suite operation; this section contains further details.

The directory $CYLC_TASK_WORK_DIR corresponds to $DATADIR/cylc-run/puma-aa168/share/<date>/<task>, where <date> is the date of this particular cycle and <task> is the task in question, e.g. nemo_cice_obsoper.

Similarly, $ROSE_DATAC is the directory containing data for this cycle time (see here for more information). This corresponds to $DATADIR/cylc-run/puma-aa168/work/<date>/<task>.

Stage 1: Obsoper

The governing script for the obsoper stage is run_nemo_cice. (This script is also used in the IAU stage.)

Firstly the directory $CYLC_TASK_WORK_DIR/fluxes is created. Flux files from $ROSE_DATAC/fluxes are softlinked into this directory (in the script this directory is called $INDIR_FLUXES; in suite.rc this points to $FLUX_DIR which itself points to $ROSE_DATAC/fluxes).

The ancillary files, background restarts, altimeter bias and observations are also softlinked into $CYLC_TASK_WORK_DIR.

The function handle_observations is then used to update the list of input files. The syntax is as follows:

handle_observations prefix namelist_file_list namelist_flags

Files of the form prefix_0*.nc are searched for and added to a list which is added to the namelist. This is done with the update_namelist script which is found in and is used as follows:

update_namelist $namelist_filename $namelist_parameter $parameter_value

This sets namelist_parameter=parameter_value in namelist_filename. Finally all of the namelist_flags are set to true.

An example usage is

handle_observations profile profbfiles ln_probn ln_t3d ln_s3d

The prefix here is profile, so files called profile_0*.nc are searched for. namelist_file_list is profbfiles and the three namelist_flags are set to true: ln_probn, ln_t3d, ln_s3d.

Stage 2: NEMOVAR

Stage 3: IAU

Example suite usage

Default setup

The default setup is to run for two days with one day of cycling, starting on 1/1/11. Much more information can be found here:

Note that for this particular date, no profile data are available, so the "daily_observations_profile" app will fail. The NEMO observation operator can still run without this data, however, and the suite should continue to run.

Different start date, run length and input data

From the Instructions page:

To change the start date and run length, edit the START_POINT and RUN_LENGTH variables in the rose-suite.conf file. You will also need to edit the "daily_fluxes", "daily_observations" and "fixed_restarts" apps to point to the locations of the input data.