Opened 12 years ago

Closed 11 years ago

#91 closed enhancement (fixed)

archive 32-bit PP files?

Reported by: jonathan Owned by: um_support
Component: UM Model Keywords:
Cc: jeff Platform:
UM Version:

Description

Dear Jeff and Lois

I would like to suggest that it would be better to archive 32-bit PP files of the kind produced by ff2pp, rather than the 64-bit PP files in UM dump format that come out of the model. That is, I would suggest that the archive system should run ff2pp (or equivalent). There are two main reasons for this. (1) The 32-bit files are half the size, but perfectly adequate for analysis. At the Hadley Centre we have never archived the 64-bit PP files and it has never been a problem for analysis. (2) The non-UM format is easier to process in analysis software. In particular, this is the format supported by the Met Office IDL library. xconv can handle both, of course. In Hadley Centre software, we use the suffix ".pp" for the non-UM 32-bit file format.

I should think that you don't want to change the HPCx archive system now, but maybe you could consider this for HECToR?

Thanks. Best wishes

Jonathan

Change History (3)

comment:1 Changed 12 years ago by jonathan

Jeff has told me that HECToR is not going to have an archive system as such i.e. all output files will be stored on disk. (That is evidently going to require a huge amount of disk space in time.) He commented that what I suggest could be done by altering one of the scripts to run ff2pp. I agree. That is what I'd suggest. It would give a 2x saving in storage space required, which I suppose wouldn't do any harm!

comment:2 Changed 12 years ago by lois

  • Cc jeff added

We could certainly have an 'archive' option that stores 32 bit PP data.

We have the opportunity to re-write the current UM archive mechanism to

  • change the current UMUI archive window to have more NCAS related options
  • offer the possibility of archiving to disk or to an archive, as HECToR will eventually have an archive mechanism
  • allow the archiving of 64 bit UM or 32 bit PP, using ff2pp, or even to 32 bit netdcf, using convsh

As with umpack, which compacts the files in the archive script, removing spare headers and any I/O padding added for I/O efficiency, this would only be an option for the fields files not the dump files.

It would be interesting to know why users archive their UM data and if they would be interested in these options, we will ask.

Lois

comment:3 Changed 11 years ago by lois

  • Resolution set to fixed
  • Status changed from new to closed

This is a planned option to be implemented in the HECToR archive scripts in 2009.

Note: See TracTickets for help on using tickets.