Version 4 (modified by grenville, 7 years ago) (diff)


The UM writes currently data in a format which is not most convenient nor desirable for many end users. Much effort is undertaken by the UM users to convert model output from the so-called fields-file format to CF-NetCDF, possibly via an intermediate, pp format. This conversion process is time consuming; for models generating large volumes of data, the time for conversion can be comparable to the model run time. In addition, the user can potentially be left with three data sets, the fields files, pp files and NetCDF files on various computer platforms. There is a strong case for modifying the UM source to enable direct output of CF-NetCDF. Time spent converting data formats is eliminated, data management is greatly simplified, and the creation and inclusion of metadata not possible through the format conversion route (by virtue of the fact that it is not present in the fields file but is available in the model) is opened up.

NCAS-CMS has considerable experience with software which does the format conversion externally to the model run. Jeff Cole authored xconv and convsh, which are widely used to convert fields files or pp files to NetCDF with support for the CF conventions. David Hassell is developing CF-Python which is a python implementation of the CF data model and can perform the pp to CF-NetCDF conversion. We have taken the next step and have begun working on modifications to the UM for direct CF-NetCDf output.

The project aims are

  • Directly write UM STASH output in CF-NetCDF format
  • Implement the entire UM STASH panel
  • Convert PP-file metadata to CF metadata
  • Support standard and mean PP files
  • Support NetCDF versions 3 and 4, including data compression
  • Support multiple versions of the UM

Considerable progress has been made and further effort is ongoing. Things to do include

  • Finish implementing the CF attributes (cell methods, bounds)
  • Further consider unlimited dimensions
  • Complete STASH panel options (meaning, grid subsection, timeseries)
  • Add support for mean PP files
  • Put in code so PP output works when files are re-initialised and for CRUNs
  • Add support for writing NetCDF data in I/O Server code

We recently described progress with this project at the UM User Workshop - please see the attached for details.

Attachments (1)

Download all attachments as: .zip