Opened 2 months ago

Last modified 41 hours ago

#3157 new help

a LGM suite failing at the "coupled" stage

Reported by: yb19052 Owned by: um_support
Component: UM Model Keywords:
Cc: Platform: NEXCS
UM Version: 10.7

Description

I have a suite (u-bp881) that I am trying to run for one year. The suite works okay for the first seven months (Jan-Jul), but it has failed in August.
Thus, I check the ‘job.err’ at ‘~/cylc-run/u-bp881/log/job/18500701T0000Z/coupled/NN’
There are several WARNING, but it does not show any specific messages,

Application 95495696 is crashing. ATP analysis proceeding…
Rank 1025 [Fri Jan 24 04:06:58 2020] [c0-1c1s14n2] application called MPI_Abort(MPI_COMM_WORLD, 0) - process 1025
atpAppSigHandler timed out waiting for shutdown. Re-raising signal.
_pmiu_daemon(SIGCHLD): [NID 02810] [c0-1c1s14n2] [Fri Jan 24 04:12:00 2020] PE RANK 1025 exit signal Aborted
[NID 02810] 2020-01-24 04:12:00 Apid 95495696: initiated application termination
[FAIL] run_model # return-code=137
2020-01-24T04:12:20Z CRITICAL - failed/EXIT

Next, I checked the ‘ocean.output’ file at ‘~/cylc-run/u-bp881/work/18500701T0000Z/coupled’
, but the file also does not show any errors.

How do I fix the issue?

Thanks
Kenji

Change History (6)

comment:1 Changed 2 months ago by grenville

Kenji

Did the model run correctly before you added

branches/dev/kenjiizumi/vn10.7_orbital_21k

Grenville

comment:2 Changed 2 months ago by yb19052

Hi Grenville,

I appreciate your response.

before adding 'branches/dev/kenjiizumi/vn10.7_orbital_21k' and changing atmospheric trace gas concentrations, my suite run correctly, and I got the one-year results.

Thus, I changed the orbit and gas concentrations.

Although I checked the 'ocean.output' at '/home/d03/kizumi/cylc-run/u-bp881/work/18500701T0000Z/coupled', the file interrupted suddenly,

Greenland iceshelf melting climatology (kg/s) : 0.
Greenland iceshelf melting adjusted value (kg/s) : 0.
Antarctica iceshelf melting climatology (kg/s) : -6310624.2001002589
Antarctica iceshelf melting adjusted value (kg/s) : -31311500.000000004
Greenland iceberg calving climatology (kg/s) : 2954121.6674141027
Greenland iceberg calving adjusted value (kg/s) : 13653000.
Antarctica iceberg calving climatology (kg/s) : 13374164.136348927
Antarctica iceberg calving adjusted value (kg/s) : 25618500.000000007
Greenland iceshelf melting climatology (kg/s

"ocean.output" [noeol] 15976L, 1026952C

Do you mean that this orbital files have any problems ?

Thanks
Kenji

comment:3 Changed 2 months ago by grenville

Kenji

Sounds likely, can you make changes incrementally to try to identify the cause of the failure. job.out includes:

Thermo iteration does not converge,istep1, my_task, i, j: 7358, 11, 33, 3
ice: Vertical thermo error

These come from CICE - I have never seen them before.

Grenville

comment:4 Changed 2 months ago by yb19052

Hi

Thank you for your comment.

I check the error and try to modify it.

I also test a suite, which is the CMIP6 piControl and works okay, with the LGM orbital file again.

Thanks
Kenji

comment:5 Changed 8 weeks ago by yb19052

Hi Grenville,

I read the CICE User's Guide v4.0 and I knew that my issue 'thermodynamic iteration error' is described as a common error in the manual.
'This error is written from ice_therm_vertical.F90 when the ice model temperature iteration is not con- verging in the thermodynamics. This is usually a problem with the forcing, but sometimes can be indicative of a timestep problem in the ice.'

Thus, first a kmt_file, which contains land mask information, was made for the LGM, and it works okay for the first 18 months so far.
In making the KMT file, we need to pay attention to the grid size. NEMO grid is x=362 and y=332, but the grid of the kmt is x=362, y=331.

Kenji


comment:6 Changed 41 hours ago by yb19052

Hi Grenville,
my suite, u-bp881 works good now. Thank you for your helps.
Thanks
Kenji

Note: See TracTickets for help on using tickets.