Opened 4 years ago

Closed 3 years ago

#1650 closed help (answered)

Working with UM data

Reported by: simon.tett Owned by: um_support
Component: Diagnostics Keywords:
Cc: Platform:
UM Version: 8.5

Description

Hi,

after some fighting with the UM at version 8.5 I'm ready to do some extended simulations using HadGEM3-A/GA6. My current "workflow" is to convert the PP data via ffpp and then bring the converted files to my local computer to further work on. This is how I've been working with HadCM3 when I run it on computers at Edinburgh.

Back when I worked for the Met Office I used a database system, called Camelot, to only extract the data I wanted from the tape system. (Camelot being developed by Jonathan Gregory, Emma Hibling(?) and I)

So my questions are:
1) Is there a better "workflow" than convert all the data in a file and bring it back to Edinburgh (or even JASMIN) for subsequent precessing.

2) Is there any kind of meta-data query system to allow working with a sub-set of the data?

thanks

Simon

Change History (6)

comment:1 Changed 4 years ago by grenville

Simon

At UM 8.5, there is little alternative to doing some kind of data conversion, either to pp or netcdf.

Moving data around has become a little less necessary (depending on volume) with use of the JASMIN VM on the newly launched RDF data analysis cluster (http://www.archer.ac.uk/documentation/rdf-guide/cluster.php) - you have access the Jasmin Analysis Platform on this VM. Moderate data volumes can be accommodated on the RDF, typically for the lifetime of a project. JASMIN is the appropriate place for larger volumes and has a much more powerful compute capability.

CF-python (http://cfpython.bitbucket.org/) allows you to work easily with subsets of CF-netcdf data - please contact David Hassell for advice.

Grenville

comment:2 Changed 4 years ago by simon.tett

Hi Grenville,

Thanks. Currently I do a two stage conversation. Field files to pp and then to appropriate byte ordering. Are there tools to do this in one go as part of the archiving system? Or even to netcdf…

Thanks for cluster link.. is idl with met office library available on it? Also is python and iris available on cluster?

Simon

comment:3 Changed 4 years ago by grenville

Simon

CF-python will convert UM fields files to CF-netcdf (CF-python is available on the ARCHER post processing nodes and on the jasmin vm - we'd need to double check that the latest and greatest is installed). It should be quite a simple job to get the UM archiving system to do the conversion - but we've not done that before.

UM archiving can convert fields files to pp; we generally don't worry about endianness, but if you insist on bigendian pp, that too can be done as part of the archiving - we do that for UM 6.6.3(HadGEM2) jobs.

IRIS is available on the jasmin vm.

Unfortunately, there is no idl licence at ARCHER (JASMIN at BADC does have midlrunning)

Grenville

comment:4 Changed 4 years ago by simon.tett

Hi Grenville,

What do I need to add to um to convert to bigendin pp data as part of the archiving process? That would simplify the work flow. Any tools to extract specified fields?

Simon

comment:5 Changed 4 years ago by grenville

Simon

Please change from using Jeff's archiving branch to this one

fcm:um_br/dev/grenville/vn8.5_archive-bigend-pp/src

and set

FF2PP_HECTOR = Y in the scripts inserts and modifications window in the umui.

I assume you have set up the appropriate keys for communications to the pp machines at ARCHER please see http://cms.ncas.ac.uk/wiki/Archer/NercArchiving if not.

Grenville

comment:6 Changed 3 years ago by ros

  • Resolution set to answered
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.