Opened 9 years ago

Closed 9 years ago

#657 closed help (fixed)

Model hangs at end of first month

Reported by: sws06djb Owned by: jeff
Component: UM Model Keywords:
Cc: d.j.brayshaw@…, simon.wilson@… Platform:
UM Version: 6.1

Description

Hello,

I'm trying to set up a slab ocean aquaplanet version of HadGAM (vn6.1). I'm currently working on doing the calibration run (xghia on PUMA) but everytime I run it, the model stops after one month - see the pe4 output file in the data directory:

Atm_Step: Timestep 1440
GATHER_FIELD: Non-Zero Error Code 2
GATHER_FIELD: Message: GATHER_FIELD: Error in COEX

(the model then sits on the processors doing nothing until the time expires).

I have a "working" calibration run in xgakg, but I was trying to get the STASH sorted for my "real" runs. I've also been trying to simplify some of the configuration in xghia (e.g., removing sulphur, tiling etc).

My first suspicion was that some of the STASH was causing a problem, but I've not been able to find out which fields (xghib-d are various attempts at this and all seem to fail similarly).

Please can you have a look into why the model is failing to run. If you can give me a pointer about what is happening, then that would be greatly appreciated. I need to start making progress with these runs very soon, so a quick response would be very helpful.

Thanks,

David

Change History (10)

comment:1 Changed 9 years ago by jeff

  • Owner changed from um_support to jeff
  • Status changed from new to accepted

Hi David

This looks like a problem with STASH packing. You can either try using this mod

~jwc/um/vn6.1/mods/fix_packing_le.mf77

which I think will fix the problem or try turning off packing for stash and stash climate meaning.

Jeff.

comment:2 Changed 9 years ago by sws06djb

Hi Jeff,

I've just tried the mod and unfortunately the problem still seems be there (job xghia).

In any case I've not knowingly altered the STASH packing settings since the job I had working previously… I have added quite a few new output diags but I've tried removing the ones I think "most dodgy" and that didn't fix it either.

Thanks,

David

comment:3 Changed 9 years ago by jeff

Hi David

Try turning off all the stash packing and see if that helps.

Jeff,

comment:4 Changed 9 years ago by sws06djb

Hi Jeff,

I *think* I managed to turn off all the stash packing - same result I'm afraid (xghia = no stash packing AND your modset; xghiz = no stash packing WITHOUT your modset).

Do you have any other ideas that I could try?

Thanks,

David

comment:5 Changed 9 years ago by sws06djb

Hi Jeff,

Just to keep you updated. I set another job running without any atmospheric STASH (I left a few fields in the slab ocean STASH) and its run for over a month now without a problem. So I guess its something I'm doing wrong in the atmospheric STASH section.

I'm setting some more runs going to try to narrow down which bit of STASH isn't working, but any clever suggestions would be appreciated.

David

comment:6 Changed 9 years ago by jeff

Hi David

Looking at jobs xghia,xghiz you haven't turned off all the stash packing. In window

Sub-Model Independent → Post Processing → Initialization and processing of mean & standard PP files

In the main table everything under "Packing Profile" should be set to 0.

Jeff.

comment:7 follow-up: Changed 9 years ago by sws06djb

Aha! I hadn't seen that! Thanks.

I've now rerun with packing profile set to 0 as you suggested and it works. I've also done a few more tests and it only seems that the packing needs to be turned off on the *a.pe… and *a.pf… STASH output files (run xghia has packing switched off everywhere, xghie has packing switched off in E & F only - both runs work).

Removing packing does make the files rather bigger though. Is there a fix for this now that we know what the problem is (given that the modset you recommended intially doesn't seem to work for me)?

If there isn't a fix, can you tell me if switching off packing will affect the filesize if I later convert the output to PP format or NETCDF for permanent storage?

Thank you for the help - sorry I missed the packing switches on the first attempt!

David

comment:8 in reply to: ↑ 7 Changed 9 years ago by jeff

Hi David

Removing packing does make the files rather bigger though. Is there a fix for this now that we know what the problem is (given that the modset you recommended intially doesn't seem to work for me)?

We don't really know what the problem is just that turning off packing stops it. Have you looked at the fields and do they all look ok? I would take quite a bit more work to track down the problem and fix it so packing can be switched on again.

If there isn't a fix, can you tell me if switching off packing will affect the filesize if I later convert the output to PP format or NETCDF for permanent storage?

No both PP and Netcdf stored the data unpacked so should be the same size whether UM packing is on or off.

Jeff.

comment:9 Changed 9 years ago by jeff

  • Cc simon.wilson@… added

Hi David

I'm on leave for the next 2 weeks so I've CCed Simon Wilson into this query so he can help if you have any more problems.

Jeff.

comment:10 Changed 9 years ago by jeff

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.