Opened 6 months ago

Closed 2 months ago

#3157 closed help (answered)

a LGM suite failing at the "coupled" stage

Reported by: yb19052 Owned by: um_support
Component: UM Model Keywords:
Cc: Platform: NEXCS
UM Version: 10.7


I have a suite (u-bp881) that I am trying to run for one year. The suite works okay for the first seven months (Jan-Jul), but it has failed in August.
Thus, I check the ‘job.err’ at ‘~/cylc-run/u-bp881/log/job/18500701T0000Z/coupled/NN’
There are several WARNING, but it does not show any specific messages,

Application 95495696 is crashing. ATP analysis proceeding…
Rank 1025 [Fri Jan 24 04:06:58 2020] [c0-1c1s14n2] application called MPI_Abort(MPI_COMM_WORLD, 0) - process 1025
atpAppSigHandler timed out waiting for shutdown. Re-raising signal.
_pmiu_daemon(SIGCHLD): [NID 02810] [c0-1c1s14n2] [Fri Jan 24 04:12:00 2020] PE RANK 1025 exit signal Aborted
[NID 02810] 2020-01-24 04:12:00 Apid 95495696: initiated application termination
[FAIL] run_model # return-code=137
2020-01-24T04:12:20Z CRITICAL - failed/EXIT

Next, I checked the ‘ocean.output’ file at ‘~/cylc-run/u-bp881/work/18500701T0000Z/coupled’
, but the file also does not show any errors.

How do I fix the issue?


Change History (7)

comment:1 Changed 6 months ago by grenville


Did the model run correctly before you added



comment:2 Changed 6 months ago by yb19052

Hi Grenville,

I appreciate your response.

before adding 'branches/dev/kenjiizumi/vn10.7_orbital_21k' and changing atmospheric trace gas concentrations, my suite run correctly, and I got the one-year results.

Thus, I changed the orbit and gas concentrations.

Although I checked the 'ocean.output' at '/home/d03/kizumi/cylc-run/u-bp881/work/18500701T0000Z/coupled', the file interrupted suddenly,

Greenland iceshelf melting climatology (kg/s) : 0.
Greenland iceshelf melting adjusted value (kg/s) : 0.
Antarctica iceshelf melting climatology (kg/s) : -6310624.2001002589
Antarctica iceshelf melting adjusted value (kg/s) : -31311500.000000004
Greenland iceberg calving climatology (kg/s) : 2954121.6674141027
Greenland iceberg calving adjusted value (kg/s) : 13653000.
Antarctica iceberg calving climatology (kg/s) : 13374164.136348927
Antarctica iceberg calving adjusted value (kg/s) : 25618500.000000007
Greenland iceshelf melting climatology (kg/s

"ocean.output" [noeol] 15976L, 1026952C

Do you mean that this orbital files have any problems ?


comment:3 Changed 6 months ago by grenville


Sounds likely, can you make changes incrementally to try to identify the cause of the failure. job.out includes:

Thermo iteration does not converge,istep1, my_task, i, j: 7358, 11, 33, 3
ice: Vertical thermo error

These come from CICE - I have never seen them before.


comment:4 Changed 6 months ago by yb19052


Thank you for your comment.

I check the error and try to modify it.

I also test a suite, which is the CMIP6 piControl and works okay, with the LGM orbital file again.


comment:5 Changed 5 months ago by yb19052

Hi Grenville,

I read the CICE User's Guide v4.0 and I knew that my issue 'thermodynamic iteration error' is described as a common error in the manual.
'This error is written from ice_therm_vertical.F90 when the ice model temperature iteration is not con- verging in the thermodynamics. This is usually a problem with the forcing, but sometimes can be indicative of a timestep problem in the ice.'

Thus, first a kmt_file, which contains land mask information, was made for the LGM, and it works okay for the first 18 months so far.
In making the KMT file, we need to pay attention to the grid size. NEMO grid is x=362 and y=332, but the grid of the kmt is x=362, y=331.


comment:6 Changed 4 months ago by yb19052

Hi Grenville,
my suite, u-bp881 works good now. Thank you for your helps.

comment:7 Changed 2 months ago by grenville

  • Resolution set to answered
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.