Opened 3 years ago

Closed 3 years ago

#2174 closed help (fixed)

COEX: Unable to WGDOS pack to this accuracy

Reported by: mattjbr123 Owned by: um_support
Component: UM Model Keywords: packing, stash, output, wgdos
Cc: Platform: Monsoon2
UM Version: 10.3

Description

Hello,

Running in to the above error regarding packing of output files at the moment. I get this error whenever I have any kind of packing turned on for output streams pp0 (UPA) and pp11 (UPL) in suite u-ak617.
I can move past it by turning packing off for these streams, but this is probably not particularly efficient space-wise, especially when I'm running ensembles.
There are quite a large number of outputs going into these files, so perhaps that has something to do with it?

Let me know if you have any ideas…

Cheers,
Matt

Change History (15)

comment:1 Changed 3 years ago by grenville

Matt

I've had this error when the data to be packed was bad - can you confirm that the data is OK.

Genvville

comment:2 Changed 3 years ago by mattjbr123

What's the quickest way to do this? It all looks OK in the job.out output

comment:3 Changed 3 years ago by grenville

Try running without packing on pp0, pp11

comment:4 Changed 3 years ago by mattjbr123

Have been running like this since this morning and it seems to be fine.

comment:5 Changed 3 years ago by grenville

Have you checked that the data it's creating is OK — if it can't pack, it can be that fields have strange values?

comment:6 Changed 3 years ago by mattjbr123

Not on this specific run. I have had the error before, disabled packing, and used the nudging diagnostics as well as simple U,V,T diagnostics which all seemed fine.

Is there a quick way to search for NaN's or very large magnitude values?

comment:7 Changed 3 years ago by simon

Hi,

I don't know if you've spotted this, but there's something _seriously_ wrong with your output, have a look at temp on model layers in ak617-r051i1p00000a.pl20080903, for instance.

Simon.

comment:8 Changed 3 years ago by mattjbr123

oooo yeah, that'd be doing it alright…

comment:9 Changed 3 years ago by mattjbr123

However, the fact that the model is still running fine, and that the job.out/pe_outputs are showing nothing obviously remiss make me think that it is an error in the writing of the outputs somewhere.
For example at timestep 1 the nudging routine output says:

: NUDGING_NUDGING_CTL: Typical relaxation parameter for "T" is: 0.55556E-01 0.55556E-01
: NUDGING_NUDGING_CTL: Typical UNUDGED value for "T" is: 0.308593E+03
: NUDGING_NUDGING_CTL: Typical NUDGED value for "T" is: 0.308584E+03
: NUDGING_NUDGING_CTL: Typical relaxation parameter for "U" is: 0.55556E-01 0.55556E-01
: NUDGING_NUDGING_CTL: Typical UNUDGED value for "U" is: 0.415499E+01
: NUDGING_NUDGING_CTL: Typical NUDGED value for "U" is: 0.422834E+01
: NUDGING_NUDGING_CTL: Typical relaxation parameter for "V" is: 0.55556E-01 0.55556E-01
: NUDGING_NUDGING_CTL: Typical UNUDGED value for "V" is: 0.320106E+01
: NUDGING_NUDGING_CTL: Typical NUDGED value for "V" is: 0.322991E+01

which all looks reasonable and does not correspond with what appears to be in the ppl output files…
This is the same for all the subsequent timesteps too.

comment:10 Changed 3 years ago by simon

Your dumps look OK also, so, yes, it's likely to be an issue writing the diagnostics. Have you run with these diagnostics before? If so, can you check these are correct in previous runs? If they are correct, there must be something different in the specification of the diagnostics in this model config.

comment:11 Changed 3 years ago by mattjbr123

Tracing the history of rose-app.conf shows:

  • These particular STASH codes (section 39) were originally in UPA/pp0/a.pa output every timestep (ALLTS) and at midnight (TA00H). I don't remember having any issues with this but can't remember for certain.
  • Then it seems I copied them to UPL/pp11/a.pl, and changed the ones in UPA to T6H.
  • Soon after I disabled the UPL output stream (which only contained section 39 diagnostics), and around this time disabled packing completely for the UPA output stream, which IIRC was to get round this same error (WGDOS unable to pack…).
  • A few months later after forgetting all about this I moved the section 39 diagnostics from UPA to UPL, which still had packing enabled, hence the then re-emergence of this error.

Shorter clearer summary:
Section 39 diagnostics timeline:

  • UPA ALLTS, UPA TA00H, then
  • UPA T6H, UPL ALLTS, then
  • UPL T6H, UPL ALLTS most recently

Strikethrough = Output disabled

So it seems that this problem has been around for a while and I hadn't really noticed it as all the other diagnostics I've been using (mainly those in sections 1 and 30) have been fine. It seems to have appeared when the diagnostics are output every 6hours. Maybe they are not 'designed' to be output every 6hrs?

I will try changing them back to TA00H (as I don't recall whether or not that was causing any issues at the time) and talk to people at the MO who know a bit more about the nudging routine and the diagnostics. Let me know if you have any other ideas!

Cheers,
Matt

Version 1, edited 3 years ago by mattjbr123 (previous) (next) (diff)

comment:12 Changed 3 years ago by simon

The patterns in the bad output are of the type seen which the sampling times for the diagnostics had be incorrectly configured. The diagnostic values have to be calculated and held in memory during the same timestep as when STASH samples them, otherwise STASH will copy variables which hold undefined values.

Therefore, it's likely that the bad diagnostics are indeed only valid at 0000Z and the 6hly sampling is causing the issue.

comment:13 Changed 3 years ago by mattjbr123

Thanks Simon - good to know. I spoke to some people at the Met Office who said similar things.
I will test it with TA00H instead of T6H sampling today to check.

Matt

comment:14 Changed 3 years ago by mattjbr123

Seems to be working much better now, thanks all :)

Matt

comment:15 Changed 3 years ago by simon

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.