Problem with NEMO restart files

[mvguarino]

Moved from ticket #2422 as ticket covering multiple issues and becoming hard to manage.

Hi Ros,

u-as245 ran fine for 3 cycles and then crashed with this error:

Traceback (most recent call last):
  File "./link_drivers", line 183, in <module>
    envinsts, launchcmds = _run_drivers(common_envars, mode)
  File "./link_drivers", line 66, in _run_drivers
    '(common_envars,\'%s\')' % (drivername, mode)
  File "<string>", line 1, in <module>
  File "/fs2/n02/n02/vittoria/cylc-run/u-as245/work/19200501T0000Z/coupled/nemo_driver.py", line 648, in run_driver
    exe_envar = _setup_executable(common_envar)
  File "/fs2/n02/n02/vittoria/cylc-run/u-as245/work/19200501T0000Z/coupled/nemo_driver.py", line 568, in _setup_executable
    controller_mode)
  File "/fs2/n02/n02/vittoria/cylc-run/u-as245/work/19200501T0000Z/coupled/top_controller.py", line 370, in run_controller
    nemo_dump_time)
  File "/fs2/n02/n02/vittoria/cylc-run/u-as245/work/19200501T0000Z/coupled/top_controller.py", line 248, in _setup_top_controller
    % top_dump_time % nemo_dump_time)
TypeError: not enough arguments for format string
[FAIL] run_model # return-code=1
Received signal ERR
cylc (scheduler - 2018-03-07T09:49:59Z): CRITICAL Task job script received signal ERR at 2018-03-07T09:49:59Z
cylc (scheduler - 2018-03-07T09:49:59Z): CRITICAL failed at 2018-03-07T09:49:59Z

It can’t find NEMO restart files, and indeed in /NEMOhist the last restart files written out are the 19200301 ones. However, looking at the job.out of coupled.19200401T0000Z the 19200501 restart files seem to have been produced (​http://puma.nerc.ac.uk/rose-bush/view/mvguarino/u-as245?&no_fuzzy_time=0&path=log/job/19200401T0000Z/coupled/01/job.out). Could this be a memory problem?

Thanks,

Vittoria

[ros]

Hi Vittoria,

Looks like you tried doing something with this suite since you raised this query. I presume you've changed the stash as you've now got a STASH error….

Regards,
Ros.

[mvguarino]

Hi Ros,
I didn’t touch the STASH at all so I am clueless about this error…
It will remain a mystery to me why NEMO restart files (for May and also for April! Altough the April coupled task ran fine) were not in the history directory. What I simply did yesterday was to re-run the March (and April) coupled task to get the restart files and allow the simulation to keep going. Or at least that was the plan…

Vittoria