Version 20 (modified by simon, 2 years ago) (diff) |
---|
Standalone UM system and FAMOUS
This is a standalone version of FAMOUS based on version 4.5 of the Met Office's Unified Model (UM).
The current version is 1.0.2.
Two standard FAMOUS versions are supplied, one with the MOSESI land-surface scheme, and the other with MOESEII.2. The base runids for these configurations are xhmkq and xfhcr (under robin) respectively.
It has a complete UM system but without the need for the complicated installation process. It is linux only as it comes with pre-built small executables. It also comes with prebuilt model and reconfiguration executables. If no model code changes are needed, then the model executables can be used for model integration without the need for any compilation at all on the installation machine.
As the executables are statically linked, they will run on any modern linux machine.
The system has the flexibility for local compilation, if required. This does need extra work, however. The communications library, gcom will need installing, together with a MPI system, see below for full details.
All of the standard FAMOUS input ancillaries are supplied, together with initial dumps.
The UM and FAMOUS can be installed on any linux system, from laptops to clusters. Currently the only job control software supported is the at daemon. It is possible to run with other job control systems such as Sun Grid Engine and SLURM, and an experiment system for this functionality is included, see below.
umui configurations are available for both MOSESI and MOSESII. These have been simplified as much as possible with the mods and handedits concatenated and superfluous options removed.
The package tarball is located at: /net/jasmin/users/simon/um_famous1.0.2.tgz
This document assumes the user is familiar with the UM and umui.
Prerequisites:
Running without compilation:
- Linux system
- at daemon
- ksh
(atd and ksh may not be installed by default)
Running with compilation (above plus):
- Intel fortran
- A MPI implementation
Quick start guide:
- Copy /net/jasmin/users/simon/um_famousx.x.x.tgz and unpack on the machine on which you want to run FAMOUS. The system will be unpacked into a directory called um_famous
- Edit example.profile to change the values of UMDIR and DATA_DIR, and append to your $HOME/.profile . (If $HOME/.profile doesn't exist, create it).
- Ensure that there is no other UM setup commands in .profile
- UMDIR should always have um_famous at the end of the path.
- Then
. $HOME/.profile
- On puma, copy the job xmjne (MOSESI) or xmjnf (MOSESII) from user simon, save and process. The "Submit" button DOES NOT WORK. As prebuilt executables are provided, there is no need to build the model. For testing, these model configs require no changes.
- On the installation machine type:
umsubmit_local -u pumauserid runid
You may be asked for your puma password.
- And the model should be running. Check by:
cd $DATA_DIR/runid tail -f *pe0
FAMOUS configuration
MOSESI and MOSESII
Seperate MOSESI and MOSESII.2 configurations are available. The base umui jobs can be used to build and run each of these.
umui
FAMOUS is configured via the umui as usual. The standard FAMOUS jobs have be rationalised to make their configuration as simple as possible. Details on this are below. The standard jobs are xmjne (MOSESI) or xmjnf (MOSESII) under user simon. All standard umui options are available, practically it is no different from running FAMOUS using a standard UM installation.
The "Submit" button DOES NOT WORK.
umsubmit_local -u pumauserid runid
on the local machine must be used instead.
Simplistically there are three different modes of running, increasing in complexity.
- Simplest
- Run a copy of the base job without any alterations. This will work using the default start date and dumps and is ideal for testing the installation and timing studies.
- Normal
- The standard umui jobs can be altered as usual changing items such STASH, start dumps, output directory location…, infact anything which doesn't require model recompilation.
- Full installation
- If changes to the model or reconfiguration code are required, then a full installation is needed. This requires the Intel compiler, and a MPI installation. See below.
modsets
The difference between MOSESI and MOSESII modsets are considerable. In fact, MOSESII code does not exist as part of the standard UM4.5 release and can only be configured as mods. Therefore it was decided that there should be different standard mods for each configuration. All of the various model mods for the two FAMOUS configurations are now held in 5 mods, these are:
- source_common.mod
- FORTRAN mods which are common to both MOSESI and MOSESII
- source_common_c.mod
- C mods which are common to both MOSESI and MOSESII
- source_MOSESI.mod
- MOSESI exclusive mods
- source_MOSESII.mod
- MOSESII exclusive mods
- port_end_f.mod
- mod which allows IO on little endian machine, such as Linux. UM files are natively bigendian. It was decided to keep this separate to allow easy porting of a model configuration to a bigendian machine such as IBM Power.
Similarly there mods for the reconfiguration.
- recon_common.mod
- FORTRAN mods which are common to both MOSESI and MOSESII
- recon_MOSESI.mod
- MOSESI exclusive mods
- recon_MOSESII.mod
- MOSESII exclusive mods
These mods are concatenations of the standard mods for each model. The tool $UMDIR/tools/make_mod.sh was used for this. Full details on the mods used are at the end of this documentation.
handedits
The handedits where concatenated in a similar way to the modsets. There is a single handedit for each model configuration. They are ~simon/famous/handedit_MOSESI.sh and ~simon/famous/handedit_MOSESII.sh . The tool make_handedit.sh was used to create them. In those cases where the original handedit changed parameter values which could be set in the umui, these values where set directly in the umui and that part of the handedit discarded.
Directories
An number of directories can be set by the user via environment variables. Depending on the directory type, this needs to be done either via the umui or via the user's $HOME/.profile . Certain variables have to be set in the .profile as these are required by the UM system and need to be set before the umui job is copied from puma for submission.
$HOME/.profile environment variables
Required:
- UMDIR
- Directory of unpacked tarball. This should end with um_famous.
- DATA_DIR
- Base model output directory. Note: DATA_DIR can also be set in the umui, and this will have precedence over the value in setvars.
See example.profile in $UMDIR for example settings.
Opitional:
- TMPDIR
- Temporary files, default $DATA_DIR/tmp . Do not set this to a shared filespace such as /tmp as this could cause issues if more than one person is running the UM on the machine.
- MY_OUTPUT
- Output .leave files produced as the model runs. Default $DATA_DIR/umui_out
- MPICH_DIR
- Set only if using local MPI version. See below. Default $DATA_DIR/mpich3.2
umui environment variables
- ARCHIVE_DIR
- Directory where archived data and dumps are stored. Default $DATA_DIR/um_archive/$RUNID
- FAMOUS_DATA
- Directory of input data. Only needs to be changed to the input data directories are moved from under $UMDIR.
Archiving
The model configurations uses the standard umui archiving system. Output data files are translated into PP before archiving, while the dumps are just copied. It can be turned off via the umui, if required.
Intel FORTRAN, MPI and gcom
The model tarball comes complete with a pre-built MPICH3.2 system. The default installation is configured to use this. If the model needs recompilation, or a different MPI system than MPICH3.2 is required to run the model (for example on some clusters), then the model needs to be recompiled on the local system. Firstly Intel FORTRAN needs to be installed. In theory it is possible for this FAMOUS version to use other compilers, but currently only Intel FORTRAN is supported. MPI has to be installed (if it hasn't been already). FAMOUS has been tested with MPICH3.2 but should work with other implementations. For MPICH3.2 follow the installation instructions at https://www.mpich.org/ . It needs to be configured for Intel FORTRAN for both F77 and F90. Once it has been installed, add the line
export MPI_DIR=/path/to/top/level/mpich
to your .profile, before the call to setvars, changing the path as required. If MPI is already installed, then add this line to point to the local installation.
Alternatively, if you local system uses modules, ensure that the required MPI (and compiler if needed) modules are loaded when a session is started, normally be placing the module load command in ~/.profile. Then ensure that $MPI_DIR is not set and
export MPI_MODULES=Y
is.
Next gcom (the system the model uses for multi-processor communications) needs to be compiled using MPI. Type
cd $UMDIR/gcom3.8/gcom make
The file $UMDIR/gcom3.8/gcom/lib/libgcom_buffered_mpi.a should now exist.
There is no need to recompile the reconfiguration unless there is a code change. The included executable independent of MPI.
The reconfiguration or model can now be compiled as usual. The standard compile system will use the MPI version defined in MPICH_PATH and the gcom version in $UMDIR/gcom3.8/gcom/lib.
IMPORTANT when compiling the model and reconfiguration, change the name of the output executable to $DATAW/$RUNID.exec or $DATAW/$RUNID.recon otherwise there is a danger that the standard executable under UMDIR will be overwritten.
Experimental: Cluster options.
FAMOUS normally run via the "at daemon
Changes from standard UM/FAMOUS
- Linux only.
- UM comes as a single pre-configured tarball.
- UM installation process no longer required.
- UM tools included in tarball.
- Scripts are hard coded, no script mods.
- All small executables are prebuilt as static executables.
- All mods grouped together into a small number of files.
- Compile system changed so that pre-built object files no longer required.
- All handedit file collated into a single file.
- umui jobs cleaned up.
- umui updated to set values directly which handedits overwrote. These handedits removed.
- umui system code updated.
- Dumps and ancils co-located.
- MPICH built and included with system for multi-processor models.
- Changed paths so that only minimal only two (UMDIR DATA_DIR) need changing.
- Ensured that system is flexible enough to allow UM experts to configure as they would a standard UM system.
- Archiving system fully integrated with model scripts.
- Fixed a
bugfeature in recon where environment variables cannot be used in user ancils paths.
Mod sets.
The following mod sets where used for the standard jobs:
Recon
Common mods:
abort.mod dummy.mod fix_ancil_name.mod port_conv_f.mod recon4837 recon_O.mod_xcggb timerupd_new.mod
MOSESI mods:
ctile6
MOSESII mods:
arerf406_ctile
Model
Common mods:
FHL_bugs.mod LANDTOGLOBAL PTOUV_LAND PTOUV_SEA UTOPIA_4p5_divUTw.mod a2o_specifiedCO2.mod abortfix.mod alk_2.mod_4.7.mh ask1f406 ask6f406 asm1f406 atmstep_flush.mod boundsfix_famous.mod_xcpsa boundsfix_hadocc.mod_xdbua boundsfix_nonmpp.mod boundsfix_vn4.5.mod diagsw_inlansea.mod dummy.mod fixfill3a.mod fixmeanctl.mf77 fixsolang.mod fixspin3a.mod fixstdia.mod ftj1f405 gbc0f406 gdr2f406 gdr3f406 gnuml45.mod gps0f406 gsm1f406 gsm2f406 gsm4f406.PUM gsm5f406 gsm8f406 gsm9f406 h3dbqlim inittime_info linuxf_mpp.mod long_output_names.mod meadlengths.mod medout44.mod model_fix.mod no_gcom_setopt.mf77 nomsrest ofilter_mpp.mod orbital_parameters-6.1.mod orh0f406 osy1f405_change params_in_namelist_famous.mod polescon port_conv_c.mod port_conv_f.mod port_end_c.mod self_shade_safe.mod_4.5 stash80.mod timerupd_new.mod tjnowrit tropin11.mod tuning_2.mod_4.7.mf77 vflux_drift.mod_301015
MOSESI mods:
UVTOP_SEA ccouple ctile_09_new_param_ftlfix inlansea_biogeo.mod ozone_mod-3levcustom schmidt_bounds.mod snowonice_coupling.mod stratcap.mod t20db_mod45_adb1f406
MOSESII mods:
cap_sflintsea.mod ccouple_moses2.2 co2_coupling_landfix_060912.mod ice_drift_fix icedrift_090912.mod inlansea_biogeo_safer.mod moses2.2_ctile_060912.mod namelist_famous_land_060912.mod namelist_famous_ocean_cc.mod oco2_cap_060912.mod ozone_mod-3levsane rayleigh_fric.mod snowonice_moses2.mod
These are included as part of the distribution under the $UMDIR/mods directory.