Opened 3 weeks ago

Last modified 42 hours ago

#2662 new help

Files appearing empty or incomplete

Reported by: feturner Owned by: um_support
Priority: normal Component: UM Model
Keywords: netCDF conversion Cc:
Platform: ARCHER UM Version: 4.5

Description

I've been running HadCM3 and am currently trying to convert the output files to netcdf. However, when I do this for a number of the jobs I've run, I get an error message saying some of the output files are missing or incomplete. For example, for the job xocda apparently the file xocdaa@pdp65nv is missing or incomplete. It's definitely there in my archive along with the other output files - is there any way of working out what is wrong with the file?

Thanks,
Fiona

Change History (9)

comment:1 Changed 2 weeks ago by feturner

Hello,

I've looked in to this some more this week, and have a list of jobs that have issues with files. The monthly files are there but empty.

xocda: xocdaa@pdp65nv and xocdaa@pcp65nv
xocdb: xocdba@pdp65nv and xocdba@pcp65nv
xocdc: xocdca@pdp67nv and xocdca@pcp67nv
xocdf: xocdfa@pdp65nv and xocdfa@pcp65nv
xocdg: xocdga@pdp69nv and xocdga@pcp69nv
xocdh: xocdha@pdp65nv and xocdha@pcp65nv
xocdi: xocdia@pdp65nv and xocdia@pcp65nv
xocdl: xocdla@pdp65nv and xocdla@pcp65nv
xocdm: xocdma@pdp65nv and xocdma@pcp65nv
xocdo: xocdoa@pdp65nv and xocdoa@pcp65nv
xocdp: xocdpa@pdp88nv and xocdpa@pcp88nv
xocdr: xocdra@pdp65nv and xocdra@pcp65nv
xocdt: xocdta@pdp65nv and xocdta@pcp65nv
xocdu: xocdua@pdp65nv and xocdua@pcp65nv
xocdv: xocdva@pdq02nv and xocdva@pcq02nv
xocdx: xocdxa@pdq06nv and xocdxa@pcq06nv

The years vary but it's always the same month that is empty. Is there any way of fixing this, or do I need to rerun HadCM3 for the problem years?

Thanks
Fiona

comment:2 Changed 6 days ago by willie

  • Keywords netCDF conversion added
  • UM Version set to 4.5

Hi Fiona,

What tools are you using to convert the file to netCDF? As you say, the files do exist and have valid data. So focussing on xocda, the file xocdaa@pdp65nv looks good in xconv, so the problem must lie with the conversion scheme.

Willie

comment:3 Changed 6 days ago by feturner

Hi Willie,

That's surprising to hear, when I checked the file size it said it was empty:

ARCHER-xc30> ls -l xocdaa@pdp65nv
-rw-r—r— 1 fturner n02 0 Oct 6 19:30 xocdaa@pdp65nv

and I haven't been able to open it in xconv. I'm using a python script to concert the file to netcdf, using the iris package.

Fiona

comment:4 Changed 6 days ago by willie

Hi Fiona,

I'm looking at,

-rw-r--r-- 1 fturner n02 3538944 Nov 13 11:54 /work/n02/n02/fturner/xocda/xocdaa@pdp65nv

Willie

comment:5 Changed 6 days ago by willie

Hi Fiona,

You are looking at, I think,

-rw-r--r-- 1 fturner n02 0 Oct  6 19:30 /nerc/n02/n02/fturner/archive/xocda/pcpd/xocdaa@pdp65nv

which is on the RDF after the post processing/archiving has been run. It looks like you are running the post processing/archiving outside the UM after the run has been completed.

Willie

comment:6 Changed 6 days ago by feturner

Hi Willie,

Yes, I'm talking about the files that I've archived. The files in my work directory are ones that I ran yesterday; after not getting a response from this ticket for a while I decided to try re-running the year that was corrupted and see if I could use that instead.

Fiona

comment:7 Changed 5 days ago by willie

Hi Fiona,

Please post the command you are using to do the convert and the error message here.

Willie

comment:8 Changed 5 days ago by feturner

The command is:

pd_stash_list=['m01s03i231','m01s03i232','m01s16i222','m01s03i236','m01s05i216','m01s03i245', \

'm01s00i031','m01s00i032','m01s00i023','m01s03i237','m01s00i001','m01s00i024', \
'm01s01i207','m01s02i205','m01s01i208','m01s03i230','m01s03i226','m01s03i225']

for VAR in range(len(pd_stash_list)):

filename = +data_root+'/'+expID+'/pcpd/'+expID+'a@pd*'
stash_constraint = iris.AttributeConstraint?(STASH=str(pd_stash_list[VAR]))
cube = iris.load_cube(filename, stash_constraint)
cube_name = cube[-1].standard_name
if cube[-1].standard_name == None:

cube_name = cube[-1].long_name

iris.save(cube, +data_root+'/'+expID+'/monthly/'+expID+'.'+str(cube_name)+'.monthly.nc')

and the error is:

iris.exceptions.TranslationError?: The file appears empty or incomplete: '/group_workspaces/jasmin4/bas_palaeoclim/users/fturner/xocda/pcpd/xocdaa@pdp65nv'

comment:9 Changed 42 hours ago by willie

Hi Fiona,

So this is a python script using Iris. I tried this on Monsoon with your file /work/n02/n02/fturner/xocda/xocdaa@pdp65nv and it did not give the error. If I run it on an empty file I get the error.

The work flow seems to be: run the UM and archive the files to the RDF and then send them to JASMIN. On JASMIN run the python script to do the convert. During this process the archiving of some files creates a zero length archive.

You don't have post processing switched on in xocda, so how are you doing the archiving?

Willie

Note: See TracTickets for help on using tickets.