Opened 4 years ago

Closed 3 years ago

#1753 closed help (answered)

New sea ice ancillary files causing UM to crash

Reported by: PUMA_GarryHayman Owned by: ros
Component: UM Model Keywords:
Cc: Platform: MONSooN
UM Version: 6.6.6

Description

I have an issue with creating sea ice ancillary fields.

I am in the process of undertaking a new set of HadGEM2 runs, building on a previous set of runs. In particular, the new runs will end 2012/2013; previously it was 2008. As a result, a number of the ancillary files have had to be updated (from their current dates between 2008 and 2010).

The first job (xldkn using HadGEM2-ES v6.6.3 as atmosphere-only on the IBM) crashed with a segmentation error after ~11.5 out of 15 year run (at mid June 2010). I ported, upgraded and subsequently repeated the job on the CRAY (xlrlf using HadGEM2-ES v6.6.6). This job also crashed at the same point with the same error. Looking at the leave output, it seemed that very strong vertical convection was apparent, immediately prior to the crash. Following Mohit Dalvi's suggestion, I increased the number of convection timesteps. I tested this (from 3 to 6 to 10 timesteps) but it made no difference.

Over the past month or so, I have been trying to identify the cause of the crash. I have switched off the nudging-related branches/hand edits. I created a set of annually-repeating ancillary files from the last year of the ‘old set’ of ancillary files. I used these in a job and then repeated the job, replacing subsets of the ‘old’ ancillary files with the ‘new set’. These runs suggested that the sea ice ancillary file was the cause of the problem (similar to the issues raised in previous UM tickets, e.g., #52 or #981).

I have two test jobs, the only difference being the sea-ice ancillary file:

  • xlrlh – using a single repeating annual dataset derived using xancil using the last year in the previous ancillary file (for 2008). This job runs to completion.
  • xlrli – using a single repeating annual dataset derived using xancil from the new sea ice dataset (also for the year 2008). This crashes.

I am using xancil to prepare the sea ice (and the other) ancillary files (see ancillary, job and namelist files in /projects/jules/ghayma/ancil_extend/z_UM_ancils_new). I have tried creating sea ice ancillary files with/without land-sea mask. I have reconfiguration on in the UM job.

In general, what is the correct process for creating and using new sea ice (and sea surface temperature) ancillary files?

Change History (3)

comment:1 Changed 4 years ago by ros

  • Owner changed from um_support to ros
  • Status changed from new to accepted

Ros

OK.

I think that I have found the reason why the sea-ice ancillary file was causing my jobs to crash. I had thought that the polar rows were all the same; this was not the case. I have now fixed this in my code used to process the sea-ice datasets and created a new ancillary file.

I set a couple of jobs running overnight:

  • xlrli - 4 month un-nudged run from June 2010: completed
  • xlrlg - 3 year run from 1st December 2009: run in progress (currently at May 2010)

So for now, please hold and I will advise you of the outcome of the run, one way or another.

For future reference, please let me know how to use the tickets.

Regards

Garry

comment:2 Changed 4 years ago by ros

Hi Garry,

Glad you have made some progress with this.

Regarding the helpdesk tickets. You need to login to the CMS website: http://cms.ncas.ac.uk with your PUMA_GarryHayman username and helpdesk password. If you can't remember what it is follow the "password reset link".

Click the "Helpdesk" link and you will see all your open tickets. To raise a new ticket click the "New Ticket" link.

Information on using the helpdesk can be found here: http://cms.ncas.ac.uk/wiki/CmsHelpdesk

Cheers,
Ros.

comment:3 Changed 3 years ago by ros

  • Resolution set to answered
  • Status changed from accepted to closed

Closed due to lack of activity.

Note: See TracTickets for help on using tickets.