wiki:UmFamous

Version 20 (modified by simon, 22 months ago) (diff)

—

Standalone UM system and FAMOUS

This is a standalone version of FAMOUS based on version 4.5 of the Met Office's Unified Model (UM).

The current version is 1.0.2.

Two standard FAMOUS versions are supplied, one with the MOSESI land-surface scheme, and the other with MOESEII.2. The base runids for these configurations are xhmkq and xfhcr (under robin) respectively.

It has a complete UM system but without the need for the complicated installation process. It is linux only as it comes with pre-built small executables. It also comes with prebuilt model and reconfiguration executables. If no model code changes are needed, then the model executables can be used for model integration without the need for any compilation at all on the installation machine.

As the executables are statically linked, they will run on any modern linux machine.

The system has the flexibility for local compilation, if required. This does need extra work, however. The communications library, gcom will need installing, together with a MPI system, see below for full details.

All of the standard FAMOUS input ancillaries are supplied, together with initial dumps.

The UM and FAMOUS can be installed on any linux system, from laptops to clusters. Currently the only job control software supported is the at daemon. It is possible to run with other job control systems such as Sun Grid Engine and SLURM, and an experiment system for this functionality is included, see below.

umui configurations are available for both MOSESI and MOSESII. These have been simplified as much as possible with the mods and handedits concatenated and superfluous options removed.

The package tarball is located at: /net/jasmin/users/simon/um_famous1.0.2.tgz

This document assumes the user is familiar with the UM and umui.

Prerequisites:

Running without compilation:

  • Linux system
  • at daemon
  • ksh

(atd and ksh may not be installed by default)

Running with compilation (above plus):

  • Intel fortran
  • A MPI implementation

Quick start guide:

  1. Copy /net/jasmin/users/simon/um_famousx.x.x.tgz and unpack on the machine on which you want to run FAMOUS. The system will be unpacked into a directory called um_famous
  1. Edit example.profile to change the values of UMDIR and DATA_DIR, and append to your $HOME/.profile . (If $HOME/.profile doesn't exist, create it).
  • Ensure that there is no other UM setup commands in .profile
  • UMDIR should always have um_famous at the end of the path.
  1. Then 
    . $HOME/.profile
  1. On puma, copy the job xmjne (MOSESI) or xmjnf (MOSESII) from user simon, save and process. The "Submit" button DOES NOT WORK. As prebuilt executables are provided, there is no need to build the model. For testing, these model configs require no changes.
  1. On the installation machine type: 
    umsubmit_local -u pumauserid runid

You may be asked for your puma password.

  1. And the model should be running. Check by: 
    cd $DATA_DIR/runid
tail -f *pe0

FAMOUS configuration

MOSESI and MOSESII

Seperate MOSESI and MOSESII.2 configurations are available. The base umui jobs can be used to build and run each of these.

umui

FAMOUS is configured via the umui as usual. The standard FAMOUS jobs have be rationalised to make their configuration as simple as possible. Details on this are below. The standard jobs are xmjne (MOSESI) or xmjnf (MOSESII) under user simon. All standard umui options are available, practically it is no different from running FAMOUS using a standard UM installation.

The "Submit" button DOES NOT WORK. 

umsubmit_local -u pumauserid runid

on the local machine must be used instead.

Simplistically there are three different modes of running, increasing in complexity.

Simplest
Run a copy of the base job without any alterations. This will work using the default start date and dumps and is ideal for testing the installation and timing studies.
Normal
The standard umui jobs can be altered as usual changing items such STASH, start dumps, output directory location…, infact anything which doesn't require model recompilation.
Full installation
If changes to the model or reconfiguration code are required, then a full installation is needed. This requires the Intel compiler, and a MPI installation. See below.

modsets

The difference between MOSESI and MOSESII modsets are considerable. In fact, MOSESII code does not exist as part of the standard UM4.5 release and can only be configured as mods. Therefore it was decided that there should be different standard mods for each configuration. All of the various model mods for the two FAMOUS configurations are now held in 5 mods, these are:

source_common.mod
FORTRAN mods which are common to both MOSESI and MOSESII
source_common_c.mod
C mods which are common to both MOSESI and MOSESII
source_MOSESI.mod
MOSESI exclusive mods
source_MOSESII.mod
MOSESII exclusive mods
port_end_f.mod
mod which allows IO on little endian machine, such as Linux. UM files are natively bigendian. It was decided to keep this separate to allow easy porting of a model configuration to a bigendian machine such as IBM Power.

Similarly there mods for the reconfiguration.

recon_common.mod
FORTRAN mods which are common to both MOSESI and MOSESII
recon_MOSESI.mod
MOSESI exclusive mods
recon_MOSESII.mod
MOSESII exclusive mods

These mods are concatenations of the standard mods for each model. The tool $UMDIR/tools/make_mod.sh was used for this. Full details on the mods used are at the end of this documentation.

handedits

The handedits where concatenated in a similar way to the modsets. There is a single handedit for each model configuration. They are ~simon/famous/handedit_MOSESI.sh and ~simon/famous/handedit_MOSESII.sh . The tool make_handedit.sh was used to create them. In those cases where the original handedit changed parameter values which could be set in the umui, these values where set directly in the umui and that part of the handedit discarded.

Directories

An number of directories can be set by the user via environment variables. Depending on the directory type, this needs to be done either via the umui or via the user's $HOME/.profile . Certain variables have to be set in the .profile as these are required by the UM system and need to be set before the umui job is copied from puma for submission.

$HOME/.profile environment variables

Required:

UMDIR
Directory of unpacked tarball. This should end with um_famous.
DATA_DIR
Base model output directory. Note: DATA_DIR can also be set in the umui, and this will have precedence over the value in setvars.

See example.profile in $UMDIR for example settings.

Opitional:

TMPDIR
Temporary files, default $DATA_DIR/tmp . Do not set this to a shared filespace such as /tmp as this could cause issues if more than one person is running the UM on the machine.
MY_OUTPUT
Output .leave files produced as the model runs. Default $DATA_DIR/umui_out
MPICH_DIR
Set only if using local MPI version. See below. Default $DATA_DIR/mpich3.2

umui environment variables

ARCHIVE_DIR
Directory where archived data and dumps are stored. Default $DATA_DIR/um_archive/$RUNID
FAMOUS_DATA
Directory of input data. Only needs to be changed to the input data directories are moved from under $UMDIR.

Archiving

The model configurations uses the standard umui archiving system. Output data files are translated into PP before archiving, while the dumps are just copied. It can be turned off via the umui, if required.

Intel FORTRAN, MPI and gcom

The model tarball comes complete with a pre-built MPICH3.2 system. The default installation is configured to use this. If the model needs recompilation, or a different MPI system than MPICH3.2 is required to run the model (for example on some clusters), then the model needs to be recompiled on the local system. Firstly Intel FORTRAN needs to be installed. In theory it is possible for this FAMOUS version to use other compilers, but currently only Intel FORTRAN is supported. MPI has to be installed (if it hasn't been already). FAMOUS has been tested with MPICH3.2 but should work with other implementations. For MPICH3.2 follow the installation instructions at ​https://www.mpich.org/ . It needs to be configured for Intel FORTRAN for both F77 and F90. Once it has been installed, add the line 

export MPI_DIR=/path/to/top/level/mpich

to your .profile, before the call to setvars, changing the path as required. If MPI is already installed, then add this line to point to the local installation.

Alternatively, if you local system uses modules, ensure that the required MPI (and compiler if needed) modules are loaded when a session is started, normally be placing the module load command in ~/.profile. Then ensure that $MPI_DIR is not set and 

export MPI_MODULES=Y

is.

Next gcom (the system the model uses for multi-processor communications) needs to be compiled using MPI. Type 

cd $UMDIR/gcom3.8/gcom
make

The file $UMDIR/gcom3.8/gcom/lib/libgcom_buffered_mpi.a should now exist.

There is no need to recompile the reconfiguration unless there is a code change. The included executable independent of MPI.

The reconfiguration or model can now be compiled as usual. The standard compile system will use the MPI version defined in MPICH_PATH and the gcom version in $UMDIR/gcom3.8/gcom/lib.

IMPORTANT when compiling the model and reconfiguration, change the name of the output executable to $DATAW/$RUNID.exec or $DATAW/$RUNID.recon otherwise there is a danger that the standard executable under UMDIR will be overwritten.

Experimental: Cluster options.

FAMOUS normally run via the "at daemon

Changes from standard UM/FAMOUS

  • Linux only.
  • UM comes as a single pre-configured tarball.
  • UM installation process no longer required.
  • UM tools included in tarball.
  • Scripts are hard coded, no script mods.
  • All small executables are prebuilt as static executables.
  • All mods grouped together into a small number of files.
  • Compile system changed so that pre-built object files no longer required.
  • All handedit file collated into a single file.
  • umui jobs cleaned up.
  • umui updated to set values directly which handedits overwrote. These handedits removed.
  • umui system code updated.
  • Dumps and ancils co-located.
  • MPICH built and included with system for multi-processor models.
  • Changed paths so that only minimal only two (UMDIR DATA_DIR) need changing.
  • Ensured that system is flexible enough to allow UM experts to configure as they would a standard UM system.
  • Archiving system fully integrated with model scripts.
  • Fixed a bug feature in recon where environment variables cannot be used in user ancils paths.

Mod sets.

The following mod sets where used for the standard jobs:

Recon

Common mods: 

abort.mod
dummy.mod
fix_ancil_name.mod
port_conv_f.mod
recon4837
recon_O.mod_xcggb
timerupd_new.mod

MOSESI mods: 

ctile6

MOSESII mods: 

arerf406_ctile

Model

Common mods: 

FHL_bugs.mod
LANDTOGLOBAL
PTOUV_LAND
PTOUV_SEA
UTOPIA_4p5_divUTw.mod
a2o_specifiedCO2.mod
abortfix.mod
alk_2.mod_4.7.mh
ask1f406
ask6f406
asm1f406
atmstep_flush.mod
boundsfix_famous.mod_xcpsa
boundsfix_hadocc.mod_xdbua
boundsfix_nonmpp.mod
boundsfix_vn4.5.mod
diagsw_inlansea.mod
dummy.mod
fixfill3a.mod
fixmeanctl.mf77
fixsolang.mod
fixspin3a.mod
fixstdia.mod
ftj1f405
gbc0f406
gdr2f406
gdr3f406
gnuml45.mod
gps0f406
gsm1f406
gsm2f406
gsm4f406.PUM
gsm5f406
gsm8f406
gsm9f406
h3dbqlim
inittime_info
linuxf_mpp.mod
long_output_names.mod
meadlengths.mod
medout44.mod
model_fix.mod
no_gcom_setopt.mf77
nomsrest
ofilter_mpp.mod
orbital_parameters-6.1.mod
orh0f406
osy1f405_change
params_in_namelist_famous.mod
polescon
port_conv_c.mod
port_conv_f.mod
port_end_c.mod
self_shade_safe.mod_4.5
stash80.mod
timerupd_new.mod
tjnowrit
tropin11.mod
tuning_2.mod_4.7.mf77
vflux_drift.mod_301015

MOSESI mods: 

UVTOP_SEA
ccouple
ctile_09_new_param_ftlfix
inlansea_biogeo.mod
ozone_mod-3levcustom
schmidt_bounds.mod
snowonice_coupling.mod
stratcap.mod
t20db_mod45_adb1f406

MOSESII mods: 

cap_sflintsea.mod
ccouple_moses2.2
co2_coupling_landfix_060912.mod
ice_drift_fix
icedrift_090912.mod
inlansea_biogeo_safer.mod
moses2.2_ctile_060912.mod
namelist_famous_land_060912.mod
namelist_famous_ocean_cc.mod
oco2_cap_060912.mod
ozone_mod-3levsane
rayleigh_fric.mod
snowonice_moses2.mod

These are included as part of the distribution under the $UMDIR/mods directory.