Version 5 (modified by grenville, 8 years ago) (diff)


The OpenIFS project is an initiative from ECMWF that will deliver a portable version of its Integrated Forecasting System to the academic community.

NCAS-CMS has agreed to host an OpenIFS repository on PUMA. This will allow those in the academic community with OpenIF licences to access OpenIFS and run it on HECToR. The initial installation of OpenIFS has been made into the repository.

OpenIFS currently serializes its output through a single processor. This can result in a major performance bottleneck when large volumes of data are being output which will likely be the case for high resolution OpenIFS integrations. NCAS-CMS is leading a 6-month HECToR dCSE project (NAG) in collaboration with ECMWF with the aim of implementing a more sophisticated I/O model in OpenIFS (on HECToR) in which I/O is performed in parallel and asynchronously with computation. The method is currently used by the operational IFS and uses the so-called Fields Database (FDB). OpenIFS writes data in GRIB format and for ease of use, the project will also see the implementation of MARS client on HECToR. We are investigating the installation of Metview also. The FDB maintains a metadata index which stores the location of the model fields in output files for later retrieval by the MARS client. FDB has been shown to scale well on IBM P6/GPFS architecture and we are hopeful that the same will be the case on Cray/Lustre? and the project will determine if this is the case.

The project has five work packages covering software installation and testing, model verification, development of metrics for verification, optimization an performance, and product delivery. Mark Richardson (NAG) and Glenn Carver (ECMWF) are our partners and have been working on installing the FDB software (and its dependencies) on HECToR for use in modules by the community as part of work package 1. That work is nearly completed.

We are beginning to write the scripts needed to extract the source code, mirror it to HECToR and build the model executable on the supercomputer which will be the basis of the job submission system.

The project timeline is presented in the attached file (openifs_fdb_hector.pdf).

Aug. 8th

Considerable progress has been made with this project. The OpenIFS repository has been created on PUMA and the OpenIFS code installed. Mark has installed the necessary grib libraries on HECToR and the FDB libraries under a package account for maintainability and has set up environment modules to help manage the software installation. The FDB calls have been activated in the OpenIFS code and we have successfully extracted and built the model on HECToR. Glenn has made available several models at differing resolutions for testing. We have concentrated on the t1279 model (about 25km global resolution). Several scenarios have been investigated - the results are summarized in the figure below. A baseline performance with minimal IO is established for 1 and 8 OMP threads the model scales well, especially when run with 8 OMP threads (see figure below right). OpenIFS running with is 'single-writer' output scheme shows the characteristic performance slow down as a single processor throttles the rest in order to gather and output data. Our test generates hourly output for a total of 58GB/model day; time spent performing output is a very significant portion of the total (~25% of wallclock for 1024 processors, rising to ~40% wallclock for 4096 processors). Running with the FDB scheme shows an impressive performance improvement, whereby the time taken for asynchronous multiprocessor output is virtually hidden and amounts to not more than 5% of the wallclock time for the run.

Attachments (4)

Download all attachments as: .zip