Opened 10 years ago

Closed 7 years ago

#530 closed defect (fixed)

xconv bug

Reported by: a.elvidge Owned by: jeff
Component: UM Tools Keywords: xconv
Cc: Platform: <select platform>
UM Version: 7.1

Description

Hi,

I have noticed that, when converting UM output files to netcdf using xconv, the program crashes when I attempt to convert too many fields. I am attempting to convert many (up to 4-dimensional) fields which would create an nc file of perhaps ~4gb. The program crashes - immediately closes, creating the new nc file but not writing anything to it. Decreasing the number of fields to create a 3gb nc file appears to work fine, and is what I have done in the past, but i now require all the fields in one file.

Thanks, Andy

Change History (6)

comment:1 Changed 10 years ago by jeff

  • Owner changed from um_support to jeff
  • Status changed from new to accepted

Hi Andy

What version are you using on what machine? Make sure you are using the latest version downloaded from here

http://ugamp.nerc.ac.uk/xconv/

Jeff.

comment:2 Changed 10 years ago by a.elvidge

Jeff,

I am using the version on puma, so I assume it is the most recent..? It has happened on numerous occasions, ever since I started using the UM. Its just that only now is it an issue for me. The files are under my hector account at work/n02/n02/aelvidge/xenob/. I wish to convert all the fields found under the files xenoba_pf000, xenoba_pf004….. etc, to xenoba_pf044. Dont know if you could give it a go using xconv on puma to see if you experience the same problem?

Cheers, Andy

comment:3 Changed 10 years ago by jeff

Hi Andy

Puma does not have an official xconv version as it is not meant to be used for data processing. You should either process your files on Hector or copy them to your local system and do it there.

Jeff.

comment:4 Changed 10 years ago by a.elvidge

Jeff,

Sorry, I meant hector, not puma; I am using the version on hector, which appears to be the most recent (Version 1.91). Additionally I realise now that I was wrong with the estimate of 4gb for the size of the intended nc file.. it would actually be more like 12gb. Any ideas?

Cheers, Andy

p.s. the files are now under work/n02/n02/aelvidge/xenob_040310/

comment:5 Changed 10 years ago by jeff

Hi Andy

This is not a xconv bug but a limitation on the sizes of the netcdf format. There is a variation on the classic netcdf format called 64 bit offset format, this should work for your files. You can't select this format in the current version of xconv but will be able to in the next version. I have a work in progress version of the next xconv version here

/home/n02/n02/jwc/um/xconv/xconv1.92

This is not finished yet so might not work and might have debugging print statements in it, the netcdf stuff should work okay, other stuff like GRIB2 support isn't finished. You can try this out if you want at your own risk.

To change the netcdf format go into Setup → Setup Xconv Defaults and change "Netcdf output format". The program you use to read this netcdf file will only work if it has been compiled with version 3.6 or later of the netcdf library.

Jeff.

comment:6 Changed 7 years ago by jeff

  • Platform set to <select platform>
  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.