Opened 4 months ago

Closed 2 months ago

#3226 closed help (answered)

PostProc failing due to missing .ps file on Monsoon

Reported by: jjas3 Owned by: um_support
Component: UKESM Keywords: UKESM AMIP PostProc
Cc: UKESM, AMIP, PostProc Platform: Monsoon2
UM Version: 11.2

Description

Hi CMS,

I hope everyone is staying well and managing to avoid Covid-19?

I've been running a 10 year UKESM AMIP 2055 suite (u-bs111) for a few days now but it has failed at the postproc stage for 20590101 with this error message:

[ERROR] Annual mean for year ending December 2058 not possible as only got 3 file(s):
/home/d05/josts/cylc-run/u-bs111/share/data/History_Data/bs111a.ps2058amj, /home/d05/josts/cylc-run/u-bs111/share/data/History_Data/bs111a.ps2058jas, /home/d05/josts/cylc-run/u-bs111/share/data/History_Data/bs111a.ps2058ond
[FAIL] Command Terminated

I've looked in '/home/d05/josts/cylc-run/u-bs111/share/data/History_Data' and can see the three '.ps' files but cannot see the missing bs111a.ps2058jfm file. In addition, some '.pm' files seem to have been transferred to mass (jun, jul, aug, sept).

Do you know what is going on and how best to proceed from here? This was running during the mass 'at risk' period on Monday 16th in case that could be a cause?

Many thanks in advance,
Johnny

Change History (6)

comment:1 Changed 4 months ago by grenville

Johnny

It's strange the the 20580401T0000Z postproc did not complete (there are no log files?) — that would account for the missing seasonal mean bs111a.ps2058jfm.

Grenville

comment:2 Changed 4 months ago by jjas3

Hi Grenville,

Thanks for looking into this for me and hope you and everyone else at CMS is well.

That is very odd, might it be because it was running when mass was 'at risk'? How is best to proceed from here, can I re-run that 20580401T0000Z postproc somehow?

Many thanks,
Johnny

comment:3 Changed 4 months ago by grenville

Johnny

Yes - you need to run insert a task (see https://cylc.github.io/doc/built-sphinx-single/index.html, section 15.6.3.34)

You can do this from the cylc gui — navigate to control→ insert task(s)…

set TASK-NAME.CYLCE-POINT to be postproc.20580401T0000Z

then Insert

That should work

Grenville

comment:4 Changed 4 months ago by jjas3

Hi Grenville,

Sorry, I tried to reply yesterday but I must not have added the comment! Thanks for your help.

I inserted the postproc.20580401T0000Z task as you suggested above and it ran successfully and triggered the next 3 postprocs to run too (postproc.20580701T0000Z, postproc.20581001T0000Z, postproc.20590101T0000Z).

The first two ran successfully but postproc.20590101T0000Z failed again with a slightly different error message:
[ERROR] Annual mean for year ending December 2058 not possible as only got 1 file(s):

/home/d05/josts/cylc-run/u-bs111/share/data/History_Data/bs111a.ps2058ond

Seems that re-running the above postprocs seems to have removed the .ps files that were there and has been unable to re-make them. Could this be because the .pm files have been moved to mass?

Is it best to just re-run from atmos_main.20580101T0000Z, using a similar insert method.

Many thanks,
Johnny

comment:5 Changed 4 months ago by grenville

Johnny

Is it best to just re-run from atmos_main.20580101T0000Z, using a similar insert method.

Probably simplest, but you will need to re insert the postproc tasks too. Try inserting atmos_main.20580101T0000Z and carefully check that it has worked, before reinserting the postproc task.
Alternatively, you could start a new run from 20580101T0000Z.

Grenville

comment:6 Changed 2 months ago by grenville

  • Resolution set to answered
  • Status changed from new to closed

closed through inactivity

Note: See TracTickets for help on using tickets.