Crashing with new SST and seaice ancilleries

[ggxmy]

Dear CMS,

My UM job of vn8.4+UKCA, tdyoo, crashed. The only change made from the running job, tdyon, is that this job uses new ancillaries of SST and seaice. I updated these data following the error found in my previous files (ticket #1631). I need to use monthly ancillaries because my job is nudged one with Gregorian calendar.

So I went back to xancil, loaded my saved job, changed the settings a little bit, and recreated the monthly ancillaries on Monsoon:/projects/ukca-leeds/myosh/ancils/.

-rw-r—r—. 1 myosh users 76791808 Sep 1 14:46 reynolds.sst.monthly_1981_2009_new
-rw-r—r—. 1 myosh users 76791808 Sep 1 14:46 reynolds.seaice.monthly_1981_2009_new

Unlike my older ancillaries, the new ones look very similar to the original daily data

/projects/um1/ancil/atmos/n96/orca1/sst/reynolds/1981_2009_360/v0/qrclim.sst
/projects/um1/ancil/atmos/n96/orca1/seaice/reynolds/1981_2009_360/v0/qrclim.seaice

in xconv, so I thought the new data were successfully created.

Then I copied these files to ARCHER:/work/n02/n02/masara/ancils/ and used them in the job. It crashed and /home/n02/n02/masara/output/tdyoo000.tdyoo.d15244.t171400.leave.20150901-180548 contains the following messages;

ATP Stack walkback for Rank 44 starting:
  _start@start.S:113
  __libc_start_main@libc-start.c:242
  main@flumeMain.f90:48
  um_shell_@um_shell.f90:1865
  u_model_@u_model.f90:2051
  initial_@initial.f90:4267
  up_ancil_@up_ancil.f90:2252
  ereport64$ereport_mod_@ereport_mod.f90:107
  gc_abort_@gc_abort.F90:136
  mpl_abort_@mpl_abort.F90:43
  pmpi_abort@0x1c3b71c
  PMPI_Abort@0x1c5a1ac
  MPID_Abort@0x1c83481
  abort@abort.c:92
  raise@pt-raise.c:42
ATP Stack walkback for Rank 44 done
Process died with signal 6: 'Aborted'
Forcing core dumps of ranks 44, 0
View application merged backtrace tree with: stat-view atpMergedBT.dot
You may need to: module load stat

_pmiu_daemon(SIGCHLD): [NID 00212] [c1-0c0s5n0] [Tue Sep  1 19:14:40 2015] PE RANK 120 exit signal Killed
_pmiu_daemon(SIGCHLD): [NID 00136] [c0-0c2s2n0] [Tue Sep  1 19:14:40 2015] PE RANK 61 exit signal Killed
_pmiu_daemon(SIGCHLD): [NID 00118] [c0-0c1s13n2] [Tue Sep  1 19:14:40 2015] PE RANK 24 exit signal Killed
[NID 00212] 2015-09-01 19:14:40 Apid 17238493: initiated application termination
tdyoo: Run failed
*****************************************************************
   Ending script   :   qsatmos
   Completion code :   137

 REPLANCA: UPDATE REQUIRED FOR FIELD 28
  REPLANCA - time interpolation for field  28
  time,time1,time2  227940.,  2*-17364696.
  hours,int,period  192,  1,  -1
  Information used in checking ancillary data set: position of lookup table in dataset: 0
  Position of first lookup table referring to data type  0
  Interval between lookup tables referring to data type  0  Number of steps 314
  STASH code in dataset  60   STASH code requested  24
 'start' position of lookup tables for dataset in overall lookup array  1021
 60,  24,  28
  UP_ANCIL : Error in REPLANCA.

? Error in routine: UP_ANCIL
? Error Code:   228
? Error Message: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH

Can you see the problem?

The old and new ancillaries are of exactly the same sizes and I don`t understand why model complains for the header size.

Best regards,
Masaru Yoshioka

[annette]

Masaru,

Ancillary field 28 is sea-surface temperature according to the ancil-master file:

/work/n02/n02/hum/vn8.4/ctldata/ANCILmaster/ANCILfields_A

The error message is a bit cryptic, but what it is complaining about is that the STASH code in the ancillary file doesn't match what it is expecting. The key line is:

STASH code in dataset  60   STASH code requested  24

So try regenerating your ancils with the corrected STASH code.

Annette