Opened 5 months ago

Closed 5 months ago

#2849 closed help (fixed)

walltime is too short

Reported by: zliu Owned by: willie
Component: UM Tools Keywords: xancil, mkancil, netcdf, cdo
Cc: Platform: ARCHER
UM Version: 10.7

Description

Hi,

I am trying to make my ancillary file on Archer by myself. The job is under /home/n02/n02/zliu/work/INPUTFILES/sst/work. But I found the speed of mkancil is very slow. I cannot finish my work within 48 hours. How can I solve this problem? Thanks.

zhen

Change History (8)

comment:1 Changed 5 months ago by willie

  • Component changed from UM Model to UM Tools
  • Keywords xancil, mkancil added

Hi Zhen,

Yes it does seem very slow - I suspect an error in the namelist. I ran it on the command line for a few minutes - it produces 4.2MB in 11 min. Looking at the output in xconv shows that it is trying to write STASH zero but it doesn't know the domain size. So check carefully the name list - see http://cms.ncas.ac.uk/documents/xancil/.

The PBS script seems contradictory too, specifying both serial and parallel options - see http://www.archer.ac.uk/documentation/user-guide/batch.php#sec-5.16.

Willie

comment:2 Changed 5 months ago by willie

  • Owner changed from um_support to willie
  • Status changed from new to accepted

comment:3 Changed 5 months ago by zliu

Hi Willie,

Thanks for your information.
Actually, I have done a subset of my file successfully under /home/n02/n02/zliu/work/INPUTFILES/sst/ancillary_test. I think my namelist is right.
If you look at the original file era_sst_n96e.nc under /home/n02/n02/zliu/work/INPUTFILES/sst in detail. The file is big on a daily scale from 1979 to 2018.
Could you please provide me with further help. Thanks.

zhen

comment:4 Changed 5 months ago by willie

Hi Zhen,

The two name lists only differ in file name. The output logs have been lost, so I can't tell what happened. You really need to modify workq.pbs so that it uses the right queues and directories - see comment:1 above.

Also please don't modify/move files when the ticket is open, at least not without telling us.

Willie

comment:5 Changed 5 months ago by zliu

Hi Willie,

Actually, I tried in login node as you did in your previous test. So I think it is not a problem of workq.pbs. Anyway, I will try this option.

Sorry to make you confused. But I don't think I have moved files.

In /home/n02/n02/zliu/work/INPUTFILES/sst/ancillary_test, I make the subset of files successfully without output logs (mkancil < xancil.namelist). Could you please indicate me more about the output logs you mentioned.

Thanks,
zhen

comment:6 Changed 5 months ago by willie

Hi Zhen,

If you look at the output with xconv while it is running, it appears to be trying to create an ancillary file with 14,610 time steps. It is taking 25 min for 22 time steps, so at this rate, it will take about 11.6 days to complete.

The input file has one point per day for 14,600 points.

TBC …

comment:7 Changed 5 months ago by zliu

Hi Willie,

Morning! I have solved the problem though I think it is a little weird to me. The problem is that the netcdf input is produced by ncl. I transfer them into another netcdf with cdo then I make the ancillary file successfully. It takes within one minute. And nco doesn't work for this problem. Thanks for your help.

Cheers,
zhen

comment:8 Changed 5 months ago by willie

  • Keywords mkancil, netcdf, cdo added; mkancil removed
  • Resolution set to fixed
  • Status changed from accepted to closed

Hi Zhen,

Great. Thanks for letting us know.

Note: See TracTickets for help on using tickets.