UKESM1 model run failing with no obvious error

[ajd]

Hi CMS,

I am encountering an issue with suite u-bu607, where the 'coupled' task for cycle 20480101 keeps failing (error message copied in below).

I have checked the usual places for any obvious errors (e.g. 'ocean.output') but can't find anything. Also, have tried switching on extra PE diagnostics but that switch in the model doesn't seem to work.

Any ideas on how to resolve this would be much appreciated!

Kind regards,
Andrea

Rank 676 [Mon Jun 22 10:12:50 2020] [c10-0c2s10n2] application called MPI_Abort(comm=0xC4000005, 1) - process 676

atpAppSigHandler: Back-end never delivered its pid. Re-raising signal.
atpAppSigHandler: Back-end never delivered its pid. Re-raising signal.
atpAppSigHandler: Back-end never delivered its pid. Re-raising signal.
_pmiu_daemon(SIGCHLD): [NID 02088] [c10-0c2s10n0] [Mon Jun 22 10:16:47 2020] PE RANK 590 exit signal Aborted
atpAppSigHandler: Back-end never delivered its pid. Re-raising signal.
[NID 02088] 2020-06-22 10:16:47 Apid 108675102: initiated application termination
[FAIL] run_model # return-code=137
2020-06-22T10:16:56Z CRITICAL - failed/EXIT

[grenville]

Andrea

In /home/d02/andit/cylc-run/u-bu607/work/20480101T0000Z/coupled/ocean.output:

==⇒>> : E R R O R

===========

stpctl: the zonal velocity is larger than 20 m/s

This common error when the ocean model goes unstable.

Grenville

[ajd]

Thanks Grenville - I searched for the string 'error' but forgot the space between the letters!

I restarted the model in the year before the failure with a perturbed atmospheric restart dump and that seems to have worked.

Many thanks,
Andrea

[grenville]

Hi Andrea

I'm assuming that it's all worked perfectly.

Grenville