#2367 closed help (answered)

Suite failed: Atmos Main - GET_EMFILE_REC

Reported by: s.varma13 Owned by: um_support
Priority: normal Component: UM Model
Keywords: GET_EMFILE_REC Cc:
Platform: Monsoon2 UM Version: 10.8

Description

Hi

My suite u-as619 failed nearing the end of its run with the following error:

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 493
? Error from routine: GET_EMFILE_REC
? Error message: Invalid time record 2, End-of-file reached ?
? Error from processor: 200
? Error number: 42

Should I just retriggger it?

Many thanks

Sunil

Change History (14)

comment:1 Changed 10 months ago by s.varma13

Update: 2 of my other suites also failed for the same reason so I have just retriggered them all.

Hope that was the correct thing to do.

Thanks

comment:2 Changed 10 months ago by grenville

Sunil

I'm looking at this now - the problem appears to be a mismatch between the UKCA emission data and your config.

Re-triggering won't fix that

Grenville

comment:3 Changed 10 months ago by s.varma13

Hi Grenville

Yes, the suite failed again. I am using year specific biomass burning files - could that be the issue? However, I am running 2 other simulations with the same setup. The other 2 failed respectively in 1999 (u-as927) and 2008 (u-as983). The latter I had run successfully before without any issue. However these 2 are still running after retriggering.

Thanks a lot.

Sunil

comment:4 Changed 10 months ago by grenville

Sunil

/projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/1970_2010/v2/ukca_emiss_DMS.nc only has data to 2010 (for example) - so the model will fail after that. Please check your ancillary data.

Grenville

comment:5 Changed 10 months ago by s.varma13

Hi Grenville, could you please let me know where in my suite has this been selected. I changed all the ancillaries in /roses/u-as691/app/um/rose-app.conf under ukca_em_files.

Thank you.

Sunil

comment:6 Changed 10 months ago by luke

Firstly, can I apologise for the incredibly unhelpful error message. It really should give the name of the file giving the problem!

The DMS file Grenville pointed to is the likely culprit - the error code of 493 is the minimum number of time fields it was expecting, meaning that the problematic file will have less than this, and the DMS file has 492, ending 2010-12-01. This is a rather strange date for a file (usually they are the middle of the month), but the fact that your model (I checked u-as691, which I think is the one you meant) died on 2010-11-29. While technically there is one time-point after this in the file, the time interpolation would need more after this, hence the error.

The data is in the netCDF file itself. You'll need to make a new DMS emissions file to allow the job to proceed. I'd also suggest looking at the other files to see if similar things are about to happen to them.

comment:7 Changed 10 months ago by luke

As a follow-up, for more information on netCDF emissions, see the UKCA tutorial

http://www.ukca.ac.uk/wiki/index.php/UKCA_Chemistry_and_Aerosol_vn10.9_Tutorial_5

and the latest version of the UMDP

https://code.metoffice.gov.uk/doc/um/latest/papers/umdp_084.pdf

comment:8 Changed 10 months ago by s.varma13

Thanks a lot Luke, I had changed the source of my emission files to cover me to 2015 or so I thought:/roses/u-as691/app/um/rose-app.conf under ukca_em_files. Could you please let me know where in my suite it is being directed to the incorrect files ending in 2010?

When I redirect the suite to files after 2010 do I have to run the suite from the beginning as it is now using new files?

comment:9 Changed 10 months ago by luke

Have you done a

rose suite-run --reload

just re-triggering wouldn't do anything I think.

comment:10 Changed 10 months ago by s.varma13

Sorry Luke - my existing suite before the fail had files up to 2015 but for some reason it is using files to 2010 and I do not know why. Could you please point me to the place in the suite where it is using the DMS File you and Grenville mentioned above?

Thank you.

Sunil

comment:11 Changed 10 months ago by luke

Ah - I thought that you'd only just changed the files.

You can find the files the UM is reading in the job.out file. These are from your u-as691 /home/d04/suvar/cylc-run/u-as691/log/job/20101101T0000Z/atmos_main/NN/job.out

ukca_em_dir =
ukca_em_files(  1) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/1970_2010/v2/ukca_emiss_DMS.nc
ukca_em_files(  2) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_Monoterp.nc
ukca_em_files(  3) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/aerocom/v1/ukca_emiss_SO2_nat.nc
ukca_em_files(  4) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_C2H6.nc
ukca_em_files(  5) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_C3H8.nc
ukca_em_files(  6) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_C5H8.nc
ukca_em_files(  7) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_CH4.nc
ukca_em_files(  8) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v3/ukca_emiss_CO.nc
ukca_em_files(  9) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_NO.nc
ukca_em_files( 10) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_NVOC.nc
ukca_em_files( 11) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_HCHO.nc
ukca_em_files( 12) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_MeCHO.nc
ukca_em_files( 13) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_Me2CO.nc
ukca_em_files( 14) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/1970_2010/v2/ukca_emiss_NH3.nc
ukca_em_files( 15) = /projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v2/ukca_emiss_NO_aircrft.nc
ukca_em_files( 16) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/BC_biofuel_1849_2015_time_series.nc
ukca_em_files( 17) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/BC_fossil_1849_2015_time_series.nc
ukca_em_files( 18) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/OC_biofuel_1849_2015_time_series.nc
ukca_em_files( 19) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/OC_fossil_1849_2015_time_series.nc
ukca_em_files( 20) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/SO2_high_1849_2015_time_series.nc
ukca_em_files( 21) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/SO2_low_1849_2015_time_series.nc
ukca_em_files( 22) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/BC_biomass_high_1849_2015_time_series.nc
ukca_em_files( 23) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/BC_biomass_low_1849_2015_time_series.nc
ukca_em_files( 24) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/OC_biomass_high_1849_2015_time_series.nc
ukca_em_files( 25) = /projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/OC_biomass_low_1849_2015_time_series.nc

These are set in using the ukca_em_files variable in the Section 34 - UKCA: UK Aerosols and Chemistry namelist panel of the um app. The current setting for these is:

'$UM_NETCDF_UKCAEMISS_DMS_DIR/$UM_NETCDF_UKCAEMISS_DMS_FILE','$UM_NETCDF_UKCAEMISS_MONOTP_DIR/$UM_NETCDF_UKCAEMISS_MONOTP_FILE','$UM_NETCDF_UKCAEMISS_SO2NAT_DIR/$UM_NETCDF_UKCAEMISS_SO2NAT_FILE','$UM_NETCDF_UKCAEMISS_C2H6_DIR/$UM_NETCDF_UKCAEMISS_C2H6_FILE','$UM_NETCDF_UKCAEMISS_C3H8_DIR/$UM_NETCDF_UKCAEMISS_C3H8_FILE','$UM_NETCDF_UKCAEMISS_C5H8_DIR/$UM_NETCDF_UKCAEMISS_C5H8_FILE','$UM_NETCDF_UKCAEMISS_CH4_DIR/$UM_NETCDF_UKCAEMISS_CH4_FILE','/projects/ukca-admin/inputs/ancil/gregorian/n96e/ukca_emiss/cmip5/2000/v3/ukca_emiss_CO.nc','$UM_NETCDF_UKCAEMISS_NO_DIR/$UM_NETCDF_UKCAEMISS_NO_FILE','$UM_NETCDF_UKCAEMISS_NVOC_DIR/$UM_NETCDF_UKCAEMISS_NVOC_FILE','$UM_NETCDF_UKCAEMISS_HCHO_DIR/$UM_NETCDF_UKCAEMISS_HCHO_FILE','$UM_NETCDF_UKCAEMISS_MECHO_DIR/$UM_NETCDF_UKCAEMISS_MECHO_FILE','$UM_NETCDF_UKCAEMISS_ME2CO_DIR/$UM_NETCDF_UKCAEMISS_ME2CO_FILE','$UM_NETCDF_UKCAEMISS_NH3_DIR/$UM_NETCDF_UKCAEMISS_NH3_FILE','$UM_NETCDF_UKCAEMISS_NOAIR_DIR/$UM_NETCDF_UKCAEMISS_NOAIR_FILE','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/BC_biofuel_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/BC_fossil_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/OC_biofuel_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/OC_fossil_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/SO2_high_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/SO2_low_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/BC_biomass_high_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/BC_biomass_low_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/OC_biomass_high_1849_2015_time_series.nc','/projects/ancils/cmip6/ancils/n96e/timeseries_1850-2014/AerosolChemistryEmissions/v1/OC_biomass_low_1849_2015_time_series.nc'

So the one you want to change is the '$UM_NETCDF_UKCAEMISS_DMS_DIR/$UM_NETCDF_UKCAEMISS_DMS_FILE' one (in fact the first). You can do this simply by deleting this entry and then pointing to your own.

More generally, these are set by environment variables, which are defined by the install_ancil app. This is pointing to this directory:

/projects/ukca-admin/inputs/ancil/gregorian/n96e/ancil_versions/GA7p1_AMIP

This directory contains

[12:47:00 hadlk@exvmsrose GA7p1_AMIP]$ ls -al
total 24
drwxr-xr-x. 2 hadzm ukca-admin 4096 Oct 27 16:23 ./
drwxr-xr-x. 4 hadzm ukca-admin 4096 May 19  2017 ../
-rwxr-xr-x. 1 hadzm ukca-admin 7692 Feb 27  2017 ancil_fnames*
lrwxrwxrwx. 1 hadzm mo_users     10 Oct 27 16:23 ancils -> ancil_vers*
-rwxr-xr-x. 1 hadzm ukca-admin 6194 Feb 27  2017 ancil_vers*

The ancil_vers contains the settings for the directories, e.g.

export UM_NETCDF_UKCAEMISS_DMS_DIR=$UM_UKCA_N96EDIR/ukca_emiss/cmip5/1970_2010/v2

where

export UM_UKCA_N96EDIR=${UM_UKCA_N96EDIR:-/projects/ukca-admin/inputs/ancil/gregorian/n96e}

The ancil_fnames contains the filenames, e.g.

export UM_NETCDF_UKCAEMISS_DMS_FILE=ukca_emiss_DMS.nc

So this is how the files were set.

If you want to change the files I suggest not bothering with the ancil versions/filenames unless you really want to (it can be handy when setting up an experiment set, for instance), and just put the files in as a list to the ukca_em_files varialble. If they are all in one place, and cover all the required files, then you can also make use of the ukca_em_dir variable, although this isn't necessary (as can be seen from current settings).

comment:12 Changed 10 months ago by s.varma13

Thank you Luke - that is really helpful.

I have contacted the Met Office to recommend alternate files I can use to take my run to 2014.

Can you keep this ticket open so that I can come back to it if I have any problems changing the source of the files and particularly when I restart the suite.

Have a good weekend.

Sunil

comment:13 Changed 10 months ago by s.varma13

Hi Luke

The Met Office says I need to start suite from the beginning with the change of the emission files. Please therefore close this ticket.

Many thanks for your help.

Sunil

comment:14 Changed 10 months ago by luke

  • Resolution set to answered
  • Status changed from new to closed

No problem! All the best.

Note: See TracTickets for help on using tickets.