Grid point storm

[dh023729]

Hi

As I'm running the HadGEM2-ES, the model stops as the vertical velocity w out of limit:

* Maximum vertical velocity w at timestep 1439
The column below w_limit shows number of points > 0.400E+00 m/s

pts > this timestep this run

level w_limit w_max proc and position w_max at timestep

1 0 0.462E-01 45 69.4deg W -18.8deg S 0.109E+00 1352
2 0 0.683E-01 45 69.4deg W -18.8deg S 0.135E+00 1352
3 0 0.763E-01 45 69.4deg W -18.8deg S 0.158E+00 1352
4 0 0.118E+00 95 20.6deg W 73.7deg N 0.182E+00 1351
5 0 0.178E+00 95 20.6deg W 73.7deg N 0.221E+00 539
6 0 0.238E+00 95 20.6deg W 73.7deg N 0.276E+00 533
7 0 0.288E+00 95 20.6deg W 73.7deg N 0.319E+00 1185
8 0 0.297E+00 95 20.6deg W 73.7deg N 0.384E+00 935
9 0 0.268E+00 95 20.6deg W 73.7deg N 0.455E+00 935

10 0 0.255E+00 66 155.6deg W 31.2deg N 0.501E+00 935
11 0 0.275E+00 83 26.3deg W 50.0deg N 0.538E+00 1299
12 0 0.285E+00 83 26.3deg W 50.0deg N 0.555E+00 479
13 0 0.266E+00 83 26.3deg W 50.0deg N 0.577E+00 479
14 0 0.235E+00 83 26.3deg W 50.0deg N 0.595E+00 480
15 0 0.222E+00 81 63.8deg W 47.5deg N 0.603E+00 480
16 0 0.217E+00 81 63.8deg W 47.5deg N 0.616E+00 479
17 0 0.219E+00 26 82.5deg E -28.8deg S 0.649E+00 1066
18 0 0.236E+00 26 82.5deg E -28.8deg S 0.722E+00 1066
19 0 0.251E+00 37 31.9deg E -22.5deg S 0.791E+00 1066
20 0 0.275E+00 45 69.4deg W -16.3deg S 0.844E+00 1066

I try several things to avoid this: change start dumps, and change ancillary files. But none of these methods works.

Any suggestion?

The .leave file is:
/home/n02/n02/dh023729/um/umui_out/xhgze000.xhgze.d12313.t111634.leave

Thanks,
Liang

[willie]

Hi Liang,

I think the vertical velocities involved are reasonable - this is just a mesage triggered by the velocity exceeding 0.4 m/s.

There are a number of issues to consider:

• Some STASH errors exist - these can be corrected by switching them off. Do Ctl-V on the STASH page to see the errors.
• STASH codes 309 - 339 are being set to zero. These are included via user STASH files but no entries are provided in the "initialisation of user prognostics page".
• At time step 48, the global net CO2 flux into ocean - 2nd C NaN occurs. This ought to be investigated.

I hope that helps.

regards,

Willie

[dh023729]

Hi Willie,

• Some STASH errors exist - these can be corrected by switching them off. Do Ctl-V on the STASH page to see the errors.

I've turned it off.

• At time step 48, the global net CO2 flux into ocean - 2nd C NaN occurs. This ought to be investigated.

I've confirmed with people using carbon cycle scheme, this is not a problem as the scheme been turned off.

• STASH codes 309 - 339 are being set to zero. These are included via user STASH files but no entries are provided in the "initialisation of user prognostics page".

I can nor fully understand how this occured.
In my STASH (job: xhgze), there is not any entries numbered from 309-339. I don't even find any definition in 'User STASHmaster files'. Compared to my last succeeded run, I changed ancillary files for atmosphere model and change STASH for both atm and ocean. I have no idea where this error comes from?

And, here, I assume the STASHcode 309-339 means entries in Section 0, Item 309-339. Am I right?

Many thanks,

Liang

Replying to willie:

Hi Liang,

I think the vertical velocities involved are reasonable - this is just a mesage triggered by the velocity exceeding 0.4 m/s.

There are a number of issues to consider:

• Some STASH errors exist - these can be corrected by switching them off. Do Ctl-V on the STASH page to see the errors.
• STASH codes 309 - 339 are being set to zero. These are included via user STASH files but no entries are provided in the "initialisation of user prognostics page".
• At time step 48, the global net CO2 flux into ocean - 2nd C NaN occurs. This ought to be investigated.

I hope that helps.

regards,

Willie

[willie]

Hi Liang,
Could you let me have read permission on the file called core in the work directory for xhgze please.

Regards

Willie

[dh023729]

Hi Willie,

I've changed permission.

Cheers,
Liang

[willie]

Hi Liang,

I have repeated the run with a few diagnostics switch on. The problem is that there is a segmentation fault at time step 1440. the core dump reveals it is trying to write a PPfile and it looks like it is the .pm59dec file. the 1440 corresponds to 30 days, which is the rate at which you re-initialise the PP files. So it looks like it is a STASH related problem, but I have not been able to pin point it further.

In difficult cases it is always a good idea to increase the level of diagnostic output. I did the following,

• Section 13 > Diagnostic prints - switch on "flush buffer if run fails" and set the vertical velocity test value to 10 m/s
• Output options: switch on subroutine timer diagnostics, report STASH messages and select extra diagnostics

I hope that helps.

Regards,

Willie

[dh023729]

Thanks, Willie

I just re-check the model setup in UMUI. When I press 'Check Setup' button, an error message shows up:
umui: xhgze: Errors and Warnings

Errors will be output in this window
Broken Code in window atmos_STASH_UserDiags
Variable: USERLST_A

→ Model Selection

→ Atmosphere

→ STASH

→ User-STASHmaster files. Diags, Progs & Ancills.

Verification is complete.
If an error was detected then find the window, enter and close it.
This will either generate a more informative error message or it will
result in the setting of a previously unset hidden variable.

And, as I close the 'User-STASHmaster file. Diags, Progs & Ancils' windows, several windows pop out with broken code messages:
'user-STASHmaster file <~umui/hadgem2/usertash/epflux606> includes items with a broken grid code:14. This is not supported without first providing a fix to the UM. See record RESIDUAL MNMERID.CIRC.WSTARBAR'
'user-STASHmaster file <~umui/hadgem2/usertash/epflux606> includes items with a broken grid code:14. This is not supported without first providing a fix to the UM. See record DIVERGENCE OF ELIASSEN-PALM FLUX'

Any idea about this?

Thanks,
Liang

[willie]

Hi Liang,

This is due to the epflux606 file - it is not an error, just a nuisance. If you close all the windows you should be able to carry on.

Regards

Willie