wiki:Projects/NEMOVAR/Notes

Version 53 (modified by cthomas, 4 years ago) (diff)

Notes on NEMOVAR

Introduction

Operation of the NEMOVAR program is controlled by the Rose suite labelled puma-aa164, which is a copy of puma-aa145.

Suite setup

Structure

The structure of the suite is the standard one:

  • suite.rc: suite structure and operation
  • rose-suite.info: basic suite info
  • rose-suite.conf: configuration of variables used in the suite (including paths)
  • meta/rose-meta.conf: information about the variables defined in rose-suite.conf

Variables

These are the variables that are defined in rose-suite.conf and used in suite.rc.

  • START_POINT: Start point of run
  • RUN_LENGTH: Length of run
  • CYCLE_LENGTH: Length of cycle
  • WALL_CLOCK_LIMIT: Wall clock limit
  • CALENDAR: Calendar used (default '365day')
  • CICE_COL: Number of CICE columns
  • CICE_MAXBK: Maximum number of blocks per processor for CICE
  • CICE_ROW: Number of CICE rows
  • COMPUTE_HOST: Host to use for compilation and/or model run
  • EXTRACT_HOST: Host to use for code extraction
  • NEMO_IPROC: Number of NEMO processes E-W
  • NEMO_JPROC: Number of NEMO processes N-S
  • RUNID: Prefix run ID for output files
  • BUILD: Build model? (Y/N)
  • RUN: Run model? (Y/N)
  • GROUP: User group

Apps included

The following apps are included in rose-suite.conf:

  • file:app/daily_fluxes
  • file:app/daily_observations
  • file:app/fixed_ancillary
  • file:app/fixed_restarts
  • file:app/nemovar
  • file:app/nemo_cice
  • file:app/fcm_make_nemo

By default each app points towards a location on SVN. The locations are all subdirectories of the following directory:

svn://puma.nerc.ac.uk/NEMOCICE_svn/UKMO/branches/dev/annette/r5062_standalone_apps_MONSooN/config/Rose/apps/

Each directory can be checked out locally and modified, in which case the path specified in rose-suite.conf must be changed to point to the local version.

The directories all contain a rose-app.conf file along with other files/subdirectories. Each directory is discussed in more detail below.

daily_fluxes

This corresponds to the subdirectory operational_atmospheric_forcing. This app copies the daily flux files.

  • app-interface.conf: defines IN_ADDITIONAL_DAYS and OUTDIR_FLUXES
  • rose-app.conf: sets the default command to prepare_fluxes, defines INPUT_DIR
  • bin/prepare_fluxes: Script to copy flux files from INPUT_DIR to OUTDIR_FLUXES and unzip them
  • opt/rose-app-FOAM_v13.conf: defines INPUT_DIR (this is overridden by rose-app.conf)

daily_observations

This corresponds to the subdirectory hindcast_observations. This app copies the daily observation files. Everything in the opt directory controls a different set of observations.

  • app-interface.conf: defines OUTDIR_OBSERVATIONS
  • rose-app.conf: defines INBASEDIR, OUTDIR and EXEC_DIR and sets a large number of namelist parameters
  • bin/NemoQcProg_ExtractAndProcess: Script to copy observation files in place and run the NEMO QC to file convert to 'feedback' format
  • bin/NemoScr_ObsPreProc: Script to run the NEMOQC code on operational observations
  • opt/rose-app-{altimeter,ghrsst_avhrr,ghrsst_metop,profile,seaice1,seaice2,surface}.conf: (re)defines INSUBDIR, INOBSFILE and OUTOBSFILE and sets a namelist parameter

fixed_ancillary

This corresponds to the subdirectory GO5_orca025_ancillary. This app copies ancillary files into the output directory.

  • rose-app.conf: defines a variety of environment variables (pointing to NEMO data directories) and creates softlinks. The environment variables are: NEMO_ANCIL, NEMO_FORCE, NEMO_GRIDS, NEMO_INPUTS, NEMO_EXECS and NEMO_IODEF
  • bin/copy_ancilliary: empty
  • bin/operational_ancilliary: creates a large number of softlinks

fixed_restarts

This corresponds to the subdirectory initial_restarts. This app copies the initial restarts.

  • rose-app.conf: defines RESTART_DIR
  • bin/copy_restarts: creates a link from RESTART_DIR to OUTDIR_RESTARTS

nemovar

This corresponds to the subdirectory nemovar. This app configures NEMOVAR.

  • rose-app.conf: defines a large number of namelist parameters
  • app-interface.conf: defines INDIR_ANCILLARY, INDIR_ALTBIAS, INDIR_INNOVATIONS, OUTDIR_ALTBIAS and OUTDIR_INCREMENTS
  • bin/helper_functions.sh: two helper functions
  • bin/init_nemovar.sh: initialise NEMOVAR
  • bin/run_nemovar.sh: run NEMOVAR
  • bin/run_sstbias.sh: calculate SST Bias
  • file/iodef.xml: standard NEMO file (controls fields and outputs)
  • file/xmlio_server.def: standard NEMO file
  • opt/rose-app-orca025.conf: defines a large number of namelist parameters
  • opt/rose-app-sstbias-orca025.conf: defines a large number of namelist parameters

nemo_cice

This corresponds to the subdirectory GO5_nemo_GSI6_cice. This app configures NEMO-CICE.

  • rose-app.conf: defines a large number of namelist parameters
  • app-interface.conf: defines INDIR_EXEC, INDIR_ANCILLARY, INDIR_FLUXES, INDIR_BACKGROUND_RESTARTS, obsoper variables (INDIR_OBSERVATIONS, INDIR_ALTBIAS, OUTDIR_INNOVATIONS, OUTDIR_ASSIM_BACKGROUND), iau variables (INDIR_INCREMENTS, OUTDIR_ANALYSIS_RESTARTS)
  • bin/helper_functions.sh: two helper functions
  • bin/run_nemo_cice: patched version removes handle_observations and processing profb
  • bin/update_nemo_nl: script to update namelists
  • meta/rose-meta.conf: large number of definitions including of namelist variables
  • opt/rose-app-iau.conf: namelist definitions for iau
  • opt/rose-app-obsoper.conf: namelist definitions for obsoper

fcm_make_nemo

This corresponds to the subdirectory fcm_make_puma_GO5_nemo_GSI6_cice. This app builds NEMO and related programs.

  • rose-app.conf: empty
  • file/fcm-make.cfg: links to other .cfg files
  • file/fcm-make-GO5.cfg: for compiling NEMO
  • file/fcm-make-GSI5.cfg: for compiling CICE
  • file/pwr6-xlf-opt.cfg: similar to a Makefile

NEMO-CICE

Details of the version of NEMO-CICE used will go here.

namelist

Description of the more interesting namelist variables will go here.

Suite operation

This is a rough description of what happens when the suite runs. An example cylc dependency tree can be seen on this page. The suite operates in two stages: the first stage involves compilation and setup and the second stage is where the cycling occurs.

First stage:

  • build nemo
  • run fixed_ancillary (pulls in ancillary information)
  • run fixed_restarts (the background)
  • if all successful, run nemo_cice_obsoper (the observation operator step)

Remaining stages:

  • run daily_fluxes
  • run DAILY_OBSERVATIONS
  • if all successful, run nemo_cice_obsoper (produces innovations & forecast background)
  • DAILY_OBSERVATIONS 'suicides' if nemo_cice_obsoper succeeds
  • if nemo_cice_obsoper succeeds, run nemovar (produces increments)
  • if nemovar succeeds, run nemo_cice_iau at a restart 'P1D' (incremental analysis update - produces analysis restarts)
  • if nemo_cice_iau succeeds, run nemo_cice_obsoper again

Runtime

When rose suite-run is executed a cylc task is started. This populates the $HOME/cylc-run/puma-aa164 directory with the following:

  • The app directory contains subdirectories which are listed in the original rose-suite.conf: daily_fluxes, daily_observations, fcm_make_nemo, fixed_ancillary, fixed_restarts, nemo_cice, nemovar
  • suite.rc is copied from the suite directory, adding in the definitions from rose-suite.conf This file is then processed by substituting all of the variables to make suite.rc.processed
  • Two softlinks are also created to the $DATADIR/cylc-run/puma-aa164/{work,share} directories
  • Log files appear as always, the latest of which is linked to by log
  • cylc-suite.db, cylc-suite-env, rose-suite.info
  • state directory with current states at different times
  • meta/rose-meta.conf is the same as in the suite directory

Input data

Information on the input data can be found here.

Output data

Output is placed in $DATADIR/cylc-run/puma-aa164/work and $DATADIR/cylc-run/puma-aa164/share.

As stated on the Instructions page the data take up at the moment:

  • work: 90 GB per day
  • share: 4 GB initial (fcm_make_nemo and data/ancillary) + 5.5 GB per run day

The contents of each directory are described in the next two sections.

share

The subdirectories are:

  • cycle: output from each cycle (subdirectories analysis, assim_background, background, fluxes, increments, innovations, observations, updated_altbias)
  • data: ancillary data (various .nc)
  • fcm_make_nemo: output of the build procedure (e.g. .o, .mod, .f90 files)

work

There are subdirectories for each day run over. Each subdirectory contains the rose-app-run.conf file that was present in the original directory. The additional files produced in the default setup are as follows. NNNN indicates the particular subjob number (from 0000 to 0192 in the default setup) and DDDD indicates a date.

  • daily_fluxes: nothing else in here
  • daily_observations_{altimeter,ghrsst_avhrr,ghrsst_metop,profile,seaice1,seaice2,surface}: various namelist and .nc files
  • fcm_make_nemo: the .cfg files listed above
  • fixed_ancillary: nothing else in here
  • fixed_restarts: nothing else in here
  • nemo_cice_iau: some NEMO input files, altbias_NNNN.nc, coordinates_NNNN.nc, geothermal_heating_NNNN.nc, nemo_nemovari.DDDD.nc, nemo_nemovaro_00000072_analysis_restart_NNNN.nc, nemo_nemovaro_00000072_pcbias_NNNN.nc, pcbias_NNNN.nc, restart_NNNN.nc
  • nemo_cice_obsoper: some NEMO input files, altbias_NNNN.nc, assim_background_state_DI_NNNN.nc, assim_background_state_Jb_NNNN.nc, coordinates_NNNN.nc, geothermal_heating_NNNN.nc, nemo_nemovari.DDDD.nc, nemo_nemovaro_00000072_analysis_restart_NNNN.nc, nemo_nemovaro_00000072_pcbias_NNNN.nc, pcbias_NNNN.nc, restart_NNNN.nc, seaicefb_01_fdbk_NNNN.nc, seaicefb_02_fdbk_NNNN.nc, slafb_01_fdbk_NNNN.nc, sstfb_01_fdbk_NNNN.nc, sstfb_02_fdbk_NNNN.nc, sstfb_03_fdbk_NNNN.nc
  • nemovar: some NEMO input files, altbiasout_NNNN.nc, assim_background_state_Jb_NNNN.nc, background.mld.T_NNNN.nc, normalization.lookup_NNNN.nc, orca025l75_00000001_restart_NNNN.nc, orca025l75_00000001_restart_NNNN_NNNN.nc, ratio.out_NNNN.nc

More detail

nemo_cice

There are two parts to this: nemo_cice_obsoper and nemo_cice_iau. They inherit some common features which are described below.

suite.rc

Common variables in suite.rc:

            ROSE_TASK_APP = nemo_cice
            CYCLE_LENGTH = {{CYCLE_LENGTH}} #'P1D'
            CICE_NPROC    = $NEMO_NPROC

            INDIR_EXEC=$EXEC_DIR
            INDIR_ANCILLARY=$ANCILLARY_DIR
            INDIR_FLUXES=$FLUX_DIR
            INDIR_BACKGROUND_RESTARTS=$BACKGROUND_DIR

run_nemo_cice

The script that steers the job is called run_nemo_cice. It has different sections for nemo_cice_obsoper and nemo_cice_iau.

nemo_cice_obsoper

Prerequisites

nemo_cice_obsoper depends on fcm_make_nemo, fixed_ancillary, fixed_restarts, daily_fluxes, DAILY_OBSERVATIONS.

suite.rc

In suite.rc, all of the nemo_cice variables listed above are inherited. In addition the following environment variables are defined:

            ROSE_APP_OPT_CONF_KEYS = "obsoper"
            INDIR_ALTBIAS=$ALTBIAS_DIR
            INDIR_OBSERVATIONS=$OBSERVATIONS_DIR
            OUTDIR_INNOVATIONS=$INNOVATIONS_DIR
            OUTDIR_ASSIM_BACKGROUND=$ASSIM_BACKGROUND_DIR

From the rose documentation on ROSE_APP_OPT_CONF_KEYS: Each KEY in this space delimited list switches on an optional configuration in an application. The configurations are applied in first-to-last order.

App

app/nemo_cice - pulled in from SVN

Some extracts from rose-app.conf:

[command]
default=run_nemo_cice

[env]
EXEC_UTIL=/projects/jomp/danlea/NEMO/NEMO/bin
EXEC_NEMOVAR=/projects/jomp/danlea/NEMO/NEMOVAR/bin

[namelist:namrun]
cn_exp='orca025l75'
cn_ocerst_in='restart'
cn_ocerst_out='analysis_restart'
ln_clobber=.true.
ln_dimgnnn=.false.
ln_mskland=.false.
ln_rstart=.true.
nn_chunksz=0
nn_date0=20100715
nn_istate=0
nn_it000=1
nn_itend=72
nn_leapy=1
nn_no=0
nn_rstctl=0
nn_stock=600
nn_stocklist=0
nn_write=300

Contents of app-interface.conf:

[default]
INDIR_EXEC=
INDIR_ANCILLARY=svn://fcm3/NEMO_svn/UKMO/branches/dev/chughes/r11231_standalone_apps/config/Rose/interface_types/nemo_cice_nemovar_ancillary.txt@11454
INDIR_FLUXES=svn://fcm3/NEMO_svn/UKMO/branches/dev/chughes/r11231_standalone_apps/config/Rose/interface_types/sbc_core_forcing.txt@11454
INDIR_BACKGROUND_RESTARTS=svn://fcm3/NEMO_svn/UKMO/branches/dev/chughes/r11231_standalone_apps/config/Rose/interface_types/nemo_cice_restarts.txt@11454

[obsoper]
INDIR_OBSERVATIONS=
INDIR_ALTBIAS=svn://fcm3/NEMO_svn/UKMO/branches/dev/chughes/r11231_standalone_apps/config/Rose/interface_types/altbias.txt@11458
OUTDIR_INNOVATIONS=
OUTDIR_ASSIM_BACKGROUND=

[iau]
INDIR_INCREMENTS=
OUTDIR_ANALYSIS_RESTARTS=svn://fcm3/NEMO_svn/UKMO/branches/dev/chughes/r11231_standalone_apps/config/Rose/interface_types/nemo_cice_restarts.txt@11454

Extracts from opt/rose-app-obsoper.conf:

[env]
MODE=obsoper

The key file in this step is run_nemo_cice

run_nemo_cice

Outputs

Once nemo_cice_obsoper succeeds, nemovar runs.

Example runs

Default setup

The default setup is to run for two days with one day of cycling, starting on 1/1/11. Much more information can be found here:

Note that for this particular date, no profile data are available, so the "daily_observations_profile" app will fail. The NEMO observation operator can still run without this data, however, and the suite should continue to run.

Different start date and run length

From the Instructions page:

To change the start date and run length, edit the START_POINT and RUN_LENGTH variables in the rose-suite.conf file. You will also need to edit the "daily_fluxes", "daily_observations" and "fixed_restarts" apps to point to the locations of the input data.

Adding other files

To do

  • Where is OUTDIR_RESTARTS set?
  • Define ghrsst, etc.
  • Check profile input data
  • Check output in share and work directories