Opened 7 years ago

Closed 6 years ago

#929 closed help (fixed)

convert output .p* files from from 64bit to 32bit

Reported by: emb66 Owned by: um_support
Component: UM Model Keywords:
Cc: Platform: <select platform>
UM Version: <select version>

Description

Currently running UM-UKCA v7.3 on HECToR, c.a. 50-year runs

Would prefer automatically output .p* output files as 32bit rather than 64bit (as currently outputting a lot output at high frequency, and this would save time/space).

Change History (4)

comment:1 follow-up: Changed 7 years ago by grenville

Ewa

The UM doesn't allow for outputting 32 bit .p* files - output can be packed, but that might not be set up for UKCA fields, and you can set up scripts to automatically convert output to a different format. How much data are you outputting?

Grenville

comment:2 in reply to: ↑ 1 Changed 7 years ago by emb66

Replying to grenville:

Ewa

The UM doesn't allow for outputting 32 bit .p* files - output can be packed, but that might not be set up for UKCA fields, and you can set up scripts to automatically convert output to a different format. How much data are you outputting?

Grenville

Dear Grenville,

The high-frequency output (i.e. the xxxxa.p[a-j]* files) are approx. 4.2Tb from 25 years without the compression (and I need round 50 years for each run). I could compress them afterwards, but one way or the other it's mostly that my user quota on Hector become very quickly exhausted (a day, two) before I manage to move the files to the RDF directory. And given that the manual conversion of files to 32bit takes first copying and then deleting the spare file (correct me if I'm wrong?), this operation would be less quota-constrained on RDF, not Hector. What would you suggest I do then?

Ewa

comment:3 Changed 7 years ago by grenville

Ewa

The simplest way to avoid using up your disc would be to write a small script which the model will run periodically (you can set the frequency) to move data to the RDF or to convert the data to pp and then move it to the RDF.

I have a very trivial script running in job xhmte - all it does it write a line to a file each model hour. You need to include fcm:um_br/dev/jeff/VN7.3_hector_monsoon_archiving/src and the hand edit ~jeff/umui_jobs/hand_edits/archiving_7.3. Then say you would like post processing in Post Processing→Main Switch … and enter the name of your script and the frequency you want it to run in Amtosphere→Control→Post processing..→User script release.

Hope that helps.

Grenville

comment:4 Changed 6 years ago by grenville

  • Platform set to <select platform>
  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.