#1734 closed help (fixed)

STASH item causing failure

Reported by: ggxmy Owned by: simon
Priority: highest Component: UM Model
Keywords: Cc:
Platform: ARCHER UM Version: 8.4

Description

Dear CMS,

I added quite a few STASH requests to my job teafb and the job crashed. The .leave file (/home/n02/n02/masara/output/teafb000.teafb.d15322.t113623.leave.20151118-131136) shows item 2-407 is causing a problem.

 ERROR: checksum failure in climate mean
 Section  2  item  407
 This can be due to invalid values in field, or corruption of partial sum file
 Remove or fix diagnostic, and rerun

???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error in routine: U_MODEL
? Error Code:     4
? Error Message: ACUMPS: Diagnostic error. See output for item no.
? Error generated from processor:     0
? This run generated *** warnings

The crash seems to have occurred here;

  um_shell_@um_shell.f90:1865
  u_model_@u_model.f90:3865

This is the double-call downward LW forcing. I do request its single-call counterpart (2-207) since before which has not caused a problem.

I request this for TDMPMN, DIAG and UPMEAN as other radiative diagnostics which have not caused a problem. Should I request this and all other radiative diagnostics using radiation time profile? Or should I change anything else?

Likewise, in teafc and teafe, 1-241 seems to have caused a problem leaving almost exactly the same error messages.

Please could you help me?

Thanks,
Masaru Yoshioka

Change History (9)

comment:1 Changed 21 months ago by ggxmy

I'm still having the same problem with 2-407 in my updated job teafe.

What is confusing is that this same failure is caused by different diagnostics. I removed a couple of items that caused problems, like 1-241 and 1-242, but I don't want to remove 2-407.

I heard that as long as we request the diagnostics in UPMEAN, time profile will be treated properly. If this is true the time profile should not be the problem.

Once I hit the hard limit of number of fields in STASH request of 30000. In teafe, it is within the limit with a scant margin.

Masaru

comment:2 Changed 21 months ago by ggxmy

I removed the request of 2-407 and ran the job teafg. It ran for a while and crashed at time step 2232, which is the end of a month in Gregorian calendar which I use. The .leave file says this;

PPCTL_INIT_CLIMATE_MEANS: Opening new file /work/n02/n02/masara/um/teafg/teafga.pm2007dec on unit  27

OPEN:  File /work/n02/n02/masara/um/teafg/teafga.pm2007dec Created on Unit 27
IO: Open: /work/n02/n02/masara/um/teafg/teafga.pm2007dec on unit  27
PPCTL_INIT_CLIMATE_MEANS: Initialising new file on unit  27
IO: Open: /work/n02/n02/masara/um/teafg/teafga.pm2007dec on unit  27
 ERROR: UM limit of  30000  fields exceed
 Use VERIFY DIAGNOSTICS in UMUI
 See HELP in UMUI STASH panel

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!???!!!?
? Error in routine: U_MODEL
? Error Code:     4
? Error Message:  MEANDIAG: Exceed fields limit
? Error generated from processor:     0
? This run generated *** warnings
????????????????????????????????????????????????????????????????????????????????

But VERIFY DIAGNOSTICS in UMUI shows this;

29574 fields in Climate mean Period_1
29574 fields in Climate mean Period_2
29574 fields in Climate mean Period_3
29574 fields in Climate mean Period_4

so I didn't think the limit was reached. Do you know by how much I should reduce the amount of output fields? Is the limit hit for UPMEAN or total for all output streams?

Thanks,
Masaru

comment:3 follow-up: Changed 21 months ago by simon

Hi,

The limit is set via a hand edit. You'll need to update
~mdalvi/umui_jobs/hand_edits/vn8.4/climmean_field_inc.ed
and change the values to (say) 35000

I'm investigating the other issue.

Simon.

comment:4 in reply to: ↑ 3 Changed 21 months ago by ggxmy

Replying to simon:

The limit is set via a hand edit. You'll need to update
~mdalvi/umui_jobs/hand_edits/vn8.4/climmean_field_inc.ed
and change the values to (say) 35000


Thank you Simon. That's very helpful. I made my copy of this file and increased the values.

I'm investigating the other issue.


I hope there is a simple remedy for that.

Masaru

comment:5 Changed 21 months ago by simon

Hi,

I've had a look at the code, and I cannot find any part of it which processes item 2407. 2406 and 2408 are there, but not 2407. Do you know if there is a branch associated with this STASH item, or are you aware of a job where this has worked? The model is outputting rubbish for STASH 2407, and that's why it is falling over in the Climate Meaning.

As for 1241 and 1242, the help section of STASH in the umui appears to indicate these rely on item 1223 which needs to be output at the same time.

comment:6 Changed 21 months ago by ggxmy

Thank you for looking into this. The built code for job teafe are in

ARCHER:/home/n02/n02/masara/um/teafe/umatmos/ppsrc/UM/atmosphere/ and
puma:/home/ggxmy/um/um_extracts/teafe/umatmos/src/UM/atmosphere/

Do you have an access to one of these directories?

Or otherwise could you tell me where I should look? For example where are 2406 and 2408 processed? Alternatively, if you point me the subroutines to check, I will copy them to the shared directory so you can have a look.

2407 is a (all-sky) double-call LW radiative forcing at the surface. A basic question would be that if there is no code that processes 2407 then how do people evaluate radiative forcing at the surface, a very common quantity considered in many studies? Is there any alternative to 2-407?

Thanks,
Masaru

comment:7 Changed 21 months ago by simon

The code is radiation_control/diagnostics_lw.f90 There are calls to process 2206, 2207 and 2208 and then calls to process 2406 and 2408 (using i_off as an offset (equal to 200)). There is no equivalent call for 407.

I'm afraid I don't know enough about the radiation set up to offer an alternative diagnostic.

comment:8 Changed 21 months ago by simon

  • Owner changed from um_support to simon
  • Status changed from new to assigned

comment:9 Changed 20 months ago by ggxmy

  • Resolution set to fixed
  • Status changed from assigned to closed

Resolved as #1760.

Thanks for your help!

Masaru

Note: See TracTickets for help on using tickets.