Opened 3 years ago

Closed 3 years ago

#2166 closed help (answered)

checksum failure in climate mean: 34 109

Reported by: csteadman Owned by: um_support
Component: UM Model Keywords:
Cc: luke Platform: Monsoon2
UM Version: 8.4

Description

Hello,

I have encountered the same error with seven different runs on MONSOON2. The runs have run for 3-7 years, and have stopped, with the following message in the leave file:

ERROR: checksum failure in climate mean Section 34 item 109 This can be due to invalid values in field, or corruption of partial sum file Remove or fix diagnostic, and rerun

???????????????????????????????????????????????????????????????????????????????? ???!!!???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!???!!!?

? Error in routine: U_MODEL

? Error Code: 4

? Error Message: ACUMPS: Diagnostic error. See output for item no.

? Error generated from processor: 0

? This run generated * warnings ????????????????????????????????????????????????????????????????????????????????

Diagnostic 34 item 109 corresponds to black carbon.

I am running UKCA at UM vn8.4. Most of the runs I've submitted involve a new ocean-atmosphere exchange scheme, but at least one of the runs that had this same error did not use code with that scheme, so I do not believe the scheme is causing the problem.

The jobs with this error are:

xmuvi (no exchange scheme)

xnjld,e,f

xnjki,j,k

An example of a leave file containing this error is xnjlf126.xnjlf.d17126.t193310.leave

When I look at the last two dump files in xconv (xnjlfa.da20190621_00 and xnjlfa.da20190611_00) and look at 34 109 (the diagnostic mentioned in the error message, which is Accumulation BC MMR) and 34 108 (H2SO4) I do see some odd patterns. I also see odd patterns, or empty diagnostics, looking at the last dump files for xmuvi. I could send a file with screenshots if that would be helpful.

I've tried resubmitting the jobs from the last dump but that's failed. Should I try resubmitting the jobs from a dump several months earlier? If so, should I do it with a new jobID?

Thanks for your help.

Claudia

Change History (4)

comment:1 Changed 3 years ago by grenville

Claudia

I guess you can't just switch off 34 item 109 ?

Grenville

comment:2 Changed 3 years ago by csteadman

Hi Grenville,

Thanks. I switched it off and restarted from a startdump and it's now run for an additional five months, so that seems to be working. However, I'm curious about what caused the issue. Is there a known problem with the UKCA aerosol MMR diagnostics, specifically with black carbon (34 109)?

Cheers,
Claudia

comment:3 Changed 3 years ago by grenville

Claudia

We see this problem from time to time and usually make the suggestion to switch of the appropriate stash — we know that's not ideal, but it hasn't warranted deep investigation given other pressures on resource.

It doesn't only happen for stash (34, 109).

Grenville

comment:4 Changed 3 years ago by grenville

  • Resolution set to answered
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.