Notes on NEMO/NEMOVAR cycling suite for assimilation


Operation of the NEMO/NEMOVAR assimilation code is controlled by the Rose suite labelled puma-aa184, which is based on puma-aa164 (itself a copy of puma-aa145). The purpose of this suite is to assimilate data into a high-resolution ocean model using a cycling framework.

The setup of the suite is described in Suite setup and its operation is described in Suite operation. More detail on the NEMO/NEMOVAR operation can be found in the NEMO/NEMOVAR section. Finally, tips for running different configurations are in the Example suite usage section.

Suite setup

Back to top


The structure of the suite is the standard one:

  • suite.rc: suite structure and operation
  • basic suite info
  • rose-suite.conf: configuration of variables used in the suite (including paths), picked up in suite.rc
  • meta/rose-meta.conf: information about the variables defined in rose-suite.conf

Information on Rose can be found here and a guide to cylc can be found here.

The suite itself is quite simple and acts as a springboard for many other processes to run.


These are the variables that are defined in rose-suite.conf and used in suite.rc.

  • START_POINT: Start point of run
  • RUN_LENGTH: Length of run
  • CYCLE_LENGTH: Length of assimilation cycle
  • WALL_CLOCK_LIMIT: Wall clock limit
  • CALENDAR: Calendar used (default '365day')
  • CICE_COL: Number of CICE columns
  • CICE_MAXBK: Maximum number of blocks per processor for CICE
  • CICE_ROW: Number of CICE rows
  • COMPUTE_HOST: Host to use for compilation and/or model run
  • EXTRACT_HOST: Host to use for code extraction
  • NEMO_IPROC: Number of NEMO processes E-W
  • NEMO_JPROC: Number of NEMO processes N-S
  • RUNID: Prefix run ID for output files
  • BUILD: Build model? (Y/N)
  • RUN: Run model? (Y/N)
  • GROUP: User group

The most variables which change most frequently are START_POINT, RUN_LENGTH, CYCLE_LENGTH and WALL_CLOCK_LIMIT (which needs to be larger for a longer assimilation cycle).

When assimilating large files it may also be necessary to increase the memory limit. This can be done by altering ConsumableMemory in the relevant place in suite.rc.

Apps included

The suite contains several apps which are used to perform auxiliary tasks such as retrieving satellite data for a particular day. The apps are listed in rose-suite.conf:

  • file:app/daily_fluxes
  • file:app/daily_observations
  • file:app/fixed_ancillary
  • file:app/fixed_restarts
  • file:app/nemovar
  • file:app/nemo_cice
  • file:app/fcm_make_nemo

By default each app points towards a location on SVN. The SVN locations are all subdirectories of the following directory:


Each subdirectory can be checked out locally and modified, in which case the path specified in rose-suite.conf must be changed to point to the local version.

The subdirectories all contain a rose-app.conf file along with other files/subdirectories. Each app is discussed in more detail below.

Daily fluxes

The app is called daily_fluxes and corresponds to the subdirectory operational_atmospheric_forcing. This app copies the daily flux files. For day D and cycle length C, flux files from days (D - 1) to (D + C) must be present in the directory INPUT_DIR.

  • app-interface.conf: defines IN_ADDITIONAL_DAYS and OUTDIR_FLUXES
  • rose-app.conf: sets the default command to prepare_fluxes, defines INPUT_DIR
  • bin/prepare_fluxes: Script to copy flux files from INPUT_DIR to OUTDIR_FLUXES and unzip them
  • opt/rose-app-FOAM_v13.conf: defines INPUT_DIR (this is not used)

The files in INPUT_DIR must be listed in prepare_fluxes:

set -A FILESTEMS <filestem1> <filestem2> ...

where each filestem is the stem before the particular date in the filename. For example, if the stem is flux then the full filename for 1/1/11 would be

prepare_fluxes then copies and unzips the relevant flux file.

It is important to ensure the namelist reflects the contents of these files. This part of the namelist is modified in the NEMO-CICE rose-app.conf (see later).

Daily observations

The app is called daily_observations and corresponds to the subdirectory hindcast_observations. This app copies the daily observation files for assimilation into the particular cycle. Everything in the opt directory controls a different set of observations.

  • app-interface.conf: defines OUTDIR_OBSERVATIONS
  • rose-app.conf: defines INBASEDIR, OUTDIR and EXEC_DIR and sets a large number of namelist parameters
  • bin/NemoQcProg_ExtractAndProcess: Script to copy observation files in place and run the NEMO QC to file convert to 'feedback' format
  • bin/NemoScr_ObsPreProc: Script to run the NEMOQC code on operational observations (not used in this case)
  • opt/rose-app-{altimeter,ghrsst_avhrr,ghrsst_metop,profile,seaice1,seaice2,surface}.conf: (re)defines INSUBDIR, INOBSFILE and OUTOBSFILE and sets a namelist parameter

The observations are as follows:

  • Altimeter: satellite altimetry; AVISO and/or "realtime" measurements
  • GHRSST: sea surface temperature (SST) from two satellites (AVHRR, METOP)
  • Profile: profile measurements
  • Sea ice: sea ice
  • Surface: in-situ SST

The location of these files is controlled in the relevant rose-app-*.conf file; both an input directory (INBASEDIR) and file stem (INOBSFILE) are required. Similarly to the fluxes, the filenames must contain the date. Note that profile data may be monthly instead of daily.

The script NemoQcProg_ExtractAndProcess prepares the files by copying them, unzipping them and running NemoQcProg_ExtractAndProcess.exe, which applies some quality control.

Ancillary information

The app is called fixed_ancillary and corresponds to the subdirectory GO5_orca025_ancillary. This app copies ancillary files (usually one-offs that don't change during the suite's operation) into the output directory.

  • rose-app.conf: defines a variety of environment variables (pointing to NEMO data directories) and creates softlinks. The environment variables are: NEMO_ANCIL, NEMO_FORCE, NEMO_GRIDS, NEMO_INPUTS, NEMO_EXECS and NEMO_IODEF
  • bin/copy_ancilliary: empty
  • bin/operational_ancilliary: creates a large number of softlinks

Example ancillary files include bathymetry, x/y positions, rivers, and XML configuration of NEMO.

To pull in additional files or directories, edit the rose-app.conf. This defines which files to use, e.g.


Initial restarts

The app is called fixed_restarts and corresponds to the subdirectory initial_restarts. This app copies the initial restarts which are used as the background state on the first assimilation. If the cycling starts on data D, the restarts must be present for that date. In further iterations of the cycle, restarts are taken to be the analysis from the previous iteration; see Suite operation below.

  • rose-app.conf: defines RESTART_DIR
  • bin/copy_restarts: creates a link from RESTART_DIR to OUTDIR_RESTARTS, which is set in suite.rc. RESTART_DIR contains the initial restarts.


The app is called nemovar and corresponds to the subdirectory nemovar. This app configures NEMOVAR.

  • rose-app.conf: defines a large number of namelist parameters
  • bin/ two helper functions
  • bin/ initialise NEMOVAR
  • bin/ run NEMOVAR
  • bin/ calculate SST Bias
  • file/iodef.xml: standard NEMO file (controls fields and outputs)
  • file/xmlio_server.def: standard NEMO file
  • opt/rose-app-orca025.conf: defines a large number of namelist parameters
  • opt/rose-app-sstbias-orca025.conf: defines a large number of namelist parameters

The two scripts and are very important and are discussed in more detail in the NEMO/NEMOVAR section.

The various rose-app*.conf contain some namelist parameters.


The app is called nemo_cice and corresponds to the subdirectory GO5_nemo_GSI6_cice. This app configures NEMO-CICE. There are two stages: obsoper and IAU. A lot more detail can be found in the Suite operation and NEMO/NEMOVAR sections.

  • rose-app.conf: defines a large number of namelist parameters, including how to read in flux files
  • bin/ two helper functions
  • bin/run_nemo_cice: run the assimilation
  • bin/update_nemo_nl: script to update namelists
  • meta/rose-meta.conf: large number of definitions including of namelist variables
  • opt/rose-app-iau.conf: namelist definitions for iau
  • opt/rose-app-obsoper.conf: namelist definitions for obsoper

The run_nemo_cice script is particularly important here and is described more in the later sections.

The rose-app.conf contain some namelist variables which are defined in meta/rose-meta.conf.


The app is called fcm_make_nemo and corresponds to the subdirectory fcm_make_puma_GO5_nemo_GSI6_cice. This app builds NEMO and related programs.

  • rose-app.conf: empty
  • file/fcm-make.cfg: links to other .cfg files
  • file/fcm-make-GO5.cfg: for compiling NEMO
  • file/fcm-make-GSI5.cfg: for compiling CICE
  • file/pwr6-xlf-opt.cfg: similar to a Makefile

Suite operation

Back to top

This is a rough description of what happens when the suite runs. An example cylc dependency tree can be seen on this page. The suite operates in two parts: the first part involves compilation and setup and the second is where the cycling occurs.

First part:

  • build NEMO/CICE
  • run fixed_ancillary (pulls in ancillary information)
  • run fixed_restarts (the background)
  • if all successful, run nemo_cice_obsoper (the observation operator step)

Second part:

  • obtain daily fluxes
  • obtain daily observations
  • if all successful, run nemo_cice_obsoper
  • if nemo_cice_obsoper succeeds, run nemovar
  • if nemovar succeeds, run nemo_cice_iau
  • if nemo_cice_iau succeeds, run nemo_cice_obsoper at the next step in the cycle

The cycling occurs in the second part. This consists of three stages:

Stage 1: form the observation operator (obsoper). Produces the innovations y - H(xb).

Required input Program run Output
Ancillary NEMO-CICE obsoper Innovations
Restarts Forecast background

In the first cycle the restarts are the background; in subsequent cycles the restarts are the analysis produced in the previous cycle.

Stage 2: run NEMOVAR which minimises the cost function J. Produces the resulting increments Δx.

Required input Program run Output
Innovations NEMOVAR Increments
Forecast background

Stage 3: incremental analysis update (IAU); apply the increments to the background to produce the analysis xa.

Required input Program run Output
Ancillary NEMO-CICE obsoper Analysis (restarts)

These stages are described in more detail in the NEMO/NEMOVAR section.


When rose suite-run is executed a cylc task is started. This populates the $HOME/cylc-run/puma-aa168 directory with the following:

  • The app directory contains subdirectories which are listed in the original rose-suite.conf: daily_fluxes, daily_observations, fcm_make_nemo, fixed_ancillary, fixed_restarts, nemo_cice, nemovar. These have been discussed above in the suite setup section.
  • suite.rc is copied from the suite directory, adding in the definitions from rose-suite.conf This file is then processed by substituting all of the variables to make suite.rc.processed
  • Two softlinks are also created to the $DATADIR/cylc-run/puma-aa164/{work,share} directories
  • Log files appear as always, the latest of which is linked to by log
  • cylc-suite.db, cylc-suite-env, contain some suite information
  • state directory with current states at different times
  • meta/rose-meta.conf is the same as in the suite directory

Input data

Input data consists of various measurements (SST, altimetry, ice), fluxes, ancillary information and the initial restarts. These inputs have been discussed in the suite setup section. Some more information can be found here.

Output data

Output is placed in $DATADIR/cylc-run/puma-aa168/work and $DATADIR/cylc-run/puma-aa168/share. The contents of each directory are described in the next two sections.

Note: the files produced are subject to change…


The subdirectories are:

  • cycle: output from each cycle (subdirectories analysis, assim_background, background, fluxes, increments, innovations, observations, updated_altbias)
  • data: ancillary data (various .nc files)
  • fcm_make_nemo: output of the build procedure (e.g. .o, .mod, .f90 files)


There are subdirectories for each day run over. Each subdirectory contains the rose-app-run.conf file that was present in the original directory. The additional files produced in the default setup are as follows. NNNN indicates the particular subjob number (from 0000 to 0192 in the default setup) and DDDD indicates a date.

  • daily_fluxes: nothing else in here
  • daily_observations_{altimeter,ghrsst_avhrr,ghrsst_metop,profile,seaice1,seaice2,surface}: various namelist and .nc files
  • fcm_make_nemo: the .cfg files listed above
  • fixed_ancillary: nothing else in here
  • fixed_restarts: nothing else in here
  • nemo_cice_obsoper: some NEMO input files,,,,,,,,,,,,,,,,
  • nemovar: some NEMO input files,,,,,,,
  • nemo_cice_iau: some NEMO input files,,,,,,,,

See also the output files section below.


Back to top

The NEMO/NEMOVAR step is where the assimilation occurs. It has already been outlined in Suite operation; this section contains further details.

The directory $CYLC_TASK_WORK_DIR corresponds to $DATADIR/cylc-run/puma-aa168/share/<date>/<task>, where <date> is the date of this particular cycle and <task> is the task in question, e.g. nemo_cice_obsoper.

Similarly, $ROSE_DATAC is the directory containing data for this cycle time (see here for more information). This corresponds to $DATADIR/cylc-run/puma-aa168/work/<date>/<task>.

The three stages are covered in the subsections: obsoper, NEMOVAR and IAU. The output files are discussed in more detail in the output files subsection.

Stage 1: Obsoper

The governing script for the obsoper stage is run_nemo_cice. (This script is also used in the IAU stage.) Both NEMO and CICE are configured before running them.

Configuring NEMO

Firstly the directory $CYLC_TASK_WORK_DIR/fluxes is created. Flux files from $ROSE_DATAC/fluxes are softlinked into this directory (in the script this directory is called $INDIR_FLUXES; in suite.rc this points to $FLUX_DIR which itself points to $ROSE_DATAC/fluxes).

The ancillary files, background restarts, altimeter bias and observations are also softlinked into $CYLC_TASK_WORK_DIR. For the first cycle, the initial restart files are used; in subsequent cycles the files produced by the IAU stage are used.

The function handle_observations is then used to update the list of input files in namelist. The namelist is found in $DATADIR/cylc-run/puma-aa168/work/<date>/<task>/namelist and is pointed to with the variable $NEMO_NL. The syntax is as follows:

handle_observations prefix namelist_file_list namelist_flags

Files of the form prefix_0*.nc are searched for and added to a list which is added to namelist. This is done with the update_namelist script which is found in and is used as follows:

update_namelist $namelist_filename $namelist_parameter $parameter_value

This sets namelist_parameter=parameter_value in namelist_filename. Finally all of the namelist_flags are set to true.

An example usage is

handle_observations profile profbfiles ln_probn ln_t3d ln_s3d

The prefix here is profile, so files called profile_0*.nc are searched for. namelist_file_list is profbfiles and the three namelist_flags are set to true: ln_profb, ln_t3d, ln_s3d. The following lines can be found towards the end of this file:

profbfile= ''

(NB the variables are arranged alphabetically and the declaration of profbfile happens further down.)

The coordinate and geothermal files are also linked, once per processor (192 by default).

update_namelist is used to add a few more variables to the namelist:

cn_exp "'${RUNID}o'"
ln_rstart .true.
nn_rstctl 0
nn_it000 1
nn_date0 "$(rose date -c -f %Y%m%d)"
jpni ${NEMO_IPROC}
jpnj ${NEMO_JPROC}
jpnij ${NEMO_NPROC}
nitiaufin ${TIME_STEPS_PER_DAY}

The variable $RUNID is nemo_nemovar; this therefore produces files with prefix nemo_nemovaro.

Configuring CICE

The CICE namelist is also configured, called $DATADIR/cylc-run/puma-aa168/work/<date>/<task>/ice_in:

days_per_year ${DAYS_IN_YEAR}
history_file "'${RUNID}i.${CICE_HISTFREQ_N}${CICE_HISTFREQ}'"
ice_ic "'cice_restart.dat'"
incond_file "'${RUNID}i_ic'"
restart .true.
restart_file "'cice_restart.dat'"
use_leap_years ${CICE_LEAP_YEAR_FLAG}
year_init $(rose date -c -f %Y)

As before, $RUNID is nemo_nemovar so files with the prefix nemo_nemovari are produced.


After NEMO and CICE have been configured they are run (using MPI).

When the job finishes the innovations are linked to the directory $OUTDIR_INNOVATIONS (corresponding to $ROSE_DATAC/innovations).

The following links to the NEMO restarts and bias files are created:

ln -sf $CYLC_TASK_WORK_DIR/${RUNID}o_$(printf "%08d" ${TIME_STEPS_PER_CYCLE})_analysis_restart_${printf "%04d" $i}.nc 
$OUTDIR_ANALYSIS_RESTARTS/restart_$(printf "%04d" $i).nc
ln -sf $CYLC_TASK_WORK_DIR/${RUNID}o_$(printf "%08d" ${TIME_STEPS_PER_CYCLE})_pcbias_${printf "%04d" $i}.nc 
$OUTDIR_ANALYSIS_RESTARTS/pcbias_$(printf "%04d" $i).nc

Then, in $CYLC_TASK_WORK_DIR, the 192 files that have been produced for each observation type are recombined into one file. For example,

fbcomb.exe ${OUTDIR_INNOVATIONS}/ ./profb*fdbk_*.nc

is used to combine the profile files.

Finally the assimilation background files are linked as follows

ln -sf ${CYLC_TASK_WORK_DIR}/assim_background_state_Jb*.nc ${OUTDIR_ASSIM_BACKGROUND}

where ${OUTDIR_ASSIM_BACKGROUND} corresponds to $ROSE_DATAC/assim_background.

At the end of this stage, the NEMO stdout and solver stats are written to stdout.

To summarise, the obsoper step configures and runs NEMO-CICE, before copying the innovations (y - H(xb)) and assimilation background for use in the next step.

Stage 2: NEMOVAR

The two most important scripts in this case are and


Firstly is run. Links are created to the ancillary files, the coordinates and the bathymetry. Additionally the $DIR_INPUT directory is created and the innovations, the assimilation background and the altimeter bias are linked into it.

The namelist used is called namelist.nemovar. It is updated with the variable:

nn_date0 "$(rose date -c -f %Y%m%d)"

The script then calls


The coordinates, bathymetry and assim_background files are linked into $DIR_WORK.

The function interp_sd is used to interpolate between covariances in different seasons. The covariances used are for the profiles, SST, altimetry, sea ice. The current and previous seasons are determined from the $DAY variable. Weights (wgt1 and wgt2) are produced based on how far through the seasons we are. The programs ncflint and ncap are used to perform the interpolation.

${EXEC_NCO}/ncflint -O -w ${wgt1},${wgt2} -o ${file}
${EXEC_NCO}/ncap -O -s ${nco_str}" ${file} -o $file}

After the interpolation has been performed, more links are produced (e.g. Rossby radii, x/y positions).

The namelist.nemovar file is updated with links to the different observations. This is used to configure NEMOVAR.

Finally NEMOVAR itself is run.


If completes succesfully then more steps in are performed. The directory $OUTDIR_INCREMENTS is created and the increments produced in the NEMOVAR stage are combined into one file:

rebuild_nemo increments_o01 ${NEMO_NPROC}

The same is also done for the altimeter bias (altbias).

The file is one of the more important files; it contains the increments after each stage. These increments are applied to the background to create the analysis xa.

Stage 3: IAU

As with the Obsoper stage, run_nemo_cice governs the IAU stage. Many of the same actions are performed.

Configuring NEMO

Firstly the directory $CYLC_TASK_WORK_DIR/fluxes is created. Flux files from $ROSE_DATAC/fluxes are softlinked into this directory (in the script this directory is called $INDIR_FLUXES; in suite.rc this points to $FLUX_DIR which itself points to $ROSE_DATAC/fluxes).

The ancillary files, background restarts and increments (produced by NEMOVAR) are also softlinked into $CYLC_TASK_WORK_DIR.

The coordinate and geothermal files are also linked, once per processor (192 by default).

update_namelist is used to add a few more variables to the namelist (the same ones set for Obsoper).

Configuring CICE

CICE is configured in exactly the same way as during the Obsoper stage.


After NEMO and CICE have been configured they are run (using MPI).

When the job finishes the NEMO restarts and pcbias files are linked:

ln -sf $CYLC_TASK_WORK_DIR/${RUNID}o_$(printf "%08d" ${TIME_STEPS_PER_CYCLE})_analysis_restart_${printf "%04d" $i}.nc 
$OUTDIR_ANALYSIS_RESTARTS/restart_$(printf "%04d" $i).nc
ln -sf $CYLC_TASK_WORK_DIR/${RUNID}o_$(printf "%08d" ${TIME_STEPS_PER_CYCLE})_pcbias_${printf "%04d" $i}.nc 
$OUTDIR_ANALYSIS_RESTARTS/pcbias_$(printf "%04d" $i).nc

The CICE restart is also linked:

ln -sf $CYLC_TASK_WORK_DIR/cice_restart.dat.$(rose date --use-task-cycle-time --offset=P1D --format='%Y-%m-%d')-00000 

These files are used as the restarts in the next cycle.

NEMO/NEMOVAR output files

Files and links produced in the share and work directories are discussed below.


The directory is share/cycle/<date>

First cycle

Subdirectories created at start of obsoper:

  • background: LINKS to initial restarts for that day
  • fluxes: FILES of fluxes from <date-1> to <date+cycle_length>
  • observations: FILES of observations to assimilate (profile, sea ice, altimetry, SST) for that day

Subdirectories created at end of obsoper:

  • innovations: FILES of innovations for each of the observations (profile, sea ice, altimetry, SST)
  • assim_background: LINKS to work/<date>/nemo_cice_obsoper/assim_background_state_Jb*.nc

Subdirectories created at end of nemovar:

  • increments: FILE:
  • updated_altbias: FILE:

Subdirectories created at end of IAU:

  • analysis: LINKS to work/<date>/nemo_cice_iau/nemo_nemovaro*{analysis_restart,pcbias}.nc and work/<date>/nemo_cice_iau/cice_restart.dat

Subsequent cycles

Subdirectories created during the previous cycle:

  • altbias: LINK to share/cycle/<date-1>/updated_altbias/ This is created during nemovar stage of the previous cycle
  • background: LINKS to restarts from previous step share/cycle/<date-1>/analysis/{,} and share/cycle/<date-1>/analysis/cice_restart.dat This is created during the IAU stage of the previous cycle

Subdirectories created at start of obsoper:

  • fluxes: FILES of fluxes from <date-1> to <date+cycle_length>
  • observations: FILES of observations to assimilate (profile, sea ice, altimetry, SST) for that day

Subdirectories created at end of obsoper:

  • innovations: FILES of innovations for each of the observations (profile, sea ice, altimetry, SST)
  • assim_background: LINKS to work/<date>/nemo_cice_obsoper/assim_background_state_Jb*.nc


First cycle

  • nemo_cice_obsoper
    • LINKS
      • auxiliary inputs
    • FILES
      • namelist, ice_in, Ocean.conf
      • [pointed to by links from the share directory]
      • [what is this?]
      • nemo_nemovari.*
      • cice_restart.dat.<date>
      • nemo_nemovaro_00000072_{analysis_restart,pcbias}*.nc
      • {profb,slafb,sstfb,seaicefb}_01_fdbk_NNNN

  • nemovar
    • LINKS
      • auxiliary inputs
      • input directory
      • background.normalization.*.nc
    • FILES
      • data, inputs, outputs subdirectories
      • [these are used later]
      • inner_{slafb,profb,sstb}

  • nemo_cice_iau
    • LINKS
      • auxiliary inputs
    • FILES
      • nemo_nemovari_*
      • nemo_nemovaro_* [pointed to by links from the share directory]
      • cice_restart.dat.* [pointed to by links from the share directory]

Why are there two sets of nemo_nemovaro* and nemo_nemovari* files? NEMO-CICE produces them both times. A bit wasteful, but safe to delete.

Example suite usage

Back to top

Default setup

The default setup is to run for two days with one day of cycling, starting on 1/1/11. Much more information can be found here:

Note that for this particular date, no profile data are available, so the "daily_observations_profile" app will fail. The NEMO observation operator can still run without this data, however, and the suite should continue to run.

Different start date, run length and input data

From the Instructions page:

To change the start date and run length, edit the START_POINT and RUN_LENGTH variables in the rose-suite.conf file. You will also need to edit the "daily_fluxes", "daily_observations" and "fixed_restarts" apps to point to the locations of the input data.

Last modified 4 years ago Last modified on 04/08/15 17:50:25