series of simulations differeing by emissions but only some exceeding walltime limit
|Reported by:||s1374103||Owned by:||um_support|
I am using vn8.4 HadGEM3 GA4.0 UKCA CheST+GLOMAP-mode RJ4.0 on monsoon (xlsjc).
I have added a some new tracers, chemical reactions, emissions and nudged towards ERA-Interim.
At this point, the walltime limit was set to the maximum (3 hours) and the model was exceeding this, so I changed the processors.Since then I have been carrying out loads of simulations and the wallclock has not been an issue.
A few weeks ago I decided to do a series of simulations ( 2 years each) where I change just the emissions. For these series of simulations I am going to describe, note that the code/diagnostics/UMUO setup is identical, just that they are pointing towards different ancillary files (with differences in emissions)
The first job ran for 2 years perfectly fine - xncae
However, xncak, xncal are all failing due to exceeding the wallclcock limit.
The job is set to run in 1 month chunks, and it has usually been doing it in around 2 hours 20 minutes. So I'm wandering why now is it only getting to day 20 in 3 hours.
At first I thought it could be random, which is why I've submitted multiple jobs but they are all failing at the same time (around 20 days).
In addition to this, xncah failed with a different error which I cannot undestand…
sys-108 : FATAL error closing unit 6 during program termination sys-108 : FATAL error closing unit 6 during program termination sys-5 : UNRECOVERABLE error on system request Input/output error Encountered during an I/O operation on unit 6 Fortran unit 6 is connected to a sequential formatted text file: "/projects/ukca-ed/jakel/xncah/pe_output/xncah.fort6.pe74" basename: missing operand Try `basename --help' for more information. basename: missing operand Try `basename --help' for more information. ATP Stack walkback for Rank 94 starting: _start@start.S:113 firstname.lastname@example.org:242 main@flumeMain.f90:48 um_shell_@um_shell.f90:1865 u_model_@u_model.f90:2051 email@example.com:2610 firstname.lastname@example.org:3622 set_atm_pointers_@set_atm_pointers.f90:3538 ukca_init_@ukca_init.f90:656 ukca_scav_init$ukca_scavenging_mod_@ukca_scavenging_mod.f90:343 um_fort_flush_@um_fort_flush.f90:48 __flush_8@0x1d5d87a _FLUSH@0x1d5d6fa _ferr@0x1d5d2b6 _fcleanup@0x1d8970c email@example.com:92 firstname.lastname@example.org:42
Is there perhaps some differences between my jobs that I haven't noticed?
Should I change the processors again? I don't want to do this as this would mean having to repeat xncae for consistency among these emissions experiments.
Please findthe .leave files in
Any ideas would be greatly appreciated.