Opened 10 months ago

Closed 5 months ago

#3174 closed error (answered)

PE RANK 8 exit signal killed when adding a new emission file

Reported by: xuemeiwang Owned by: um_support
Component: UKCA Keywords:
Cc: Platform:
UM Version: 11.3

Description

Hi, I added a new tracer (34173) to u-br241 and a corresponding ancillary file at /projects/asci/xuewa/ancils/ukca_emiss_passive_tracer.nc. The emission file is specified in app/glm_um/opt/rose-app-ukca.conf under 'Configure ancils and initialise dump fields'. The model always fails at global model reconfiguration stage and the errors are as follows:

_pmiu_daemon(SIGCHLD): [NID 06714] [c6-2c2s14n2] [Tue Feb 4 14:41:09 2020] PE RANK 8 exit signal Killed
[FAIL] um-recon # return-code=137
2020-02-04T14:41:09Z CRITICAL - failed/EXIT

I don't quite understand how it occurs. Can you help me have a look at it?

Thanks,
Xuemei

Change History (11)

comment:1 Changed 10 months ago by grenville

Xuemei

Please set RCF_PRINTSTATUS (in the gl_um app) to 'Extra diagnostic messages' and run again - there is not enough output to point to the problem.

Grenville

comment:2 Changed 10 months ago by xuemeiwang

Hi Grenville,

Thanks!

I put both PRINT_STATUS and RCF_PRINTSTATUS to extra diagnostic messages but it still produces the same error.

Xuemei

comment:3 Changed 10 months ago by grenville

Xuemei

I can't find log files for a reconfiguration for u-br241 ?

Grenville

comment:4 Changed 10 months ago by xuemeiwang

Hi Grenville,

Sorry, I edited that suite to test if it's a file format problem. It's migrated to u-br355 which is the same as u-br241 with the same error.

Xuemei

comment:5 Changed 10 months ago by grenville

Xuemei

/home/d02/xuewa/cylc-run/u-br355/work/20140916T0000Z/glm_um_recon1/pe_output says

Ancillary File does not exist.
File : /projects/asci/xuewa/ancils/ukca_emiss_passive_tracer.nc
Stashcodes : 34173
Stashcodes : 34173
Ancil file num: 15
Ancil filename : /projects/um1/ancil/atmos/n216e/aerosol_clims/sulp/v4/qrclim.sulp70
Stash req = 359
Stash req = 360
Stash req = 361

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 10
? Error from routine: CALC_NLOOKUPS
? Error message: Ancillary files have not been found - Check output for details
? Error from processor: 0
? Error number: 2
????????????????????????????????????????????????????????????????????????????????

It appears that you have specified the passive tracer file in the wrong place, I believe it should be specify the tracer file in ukca_em_files. Are yiu following the UKCA training documentation?

Grenville
Grenville

comment:6 Changed 10 months ago by xuemeiwang

Hi Grenville,

Thanks! I was generating a netCDF file according to an existing nc ancillary but not UKCA training. I will check it again.

Xuemei

comment:7 Changed 10 months ago by xuemeiwang

Hi Grenville,

After following UKCA emission tutorial, I got an error at glm_um_fcst_000 as shown below for u-br694.

Error from routine: UKCA_MAKE_TR_MAP
Error message: Failed to set tr_lookup.

I guess it might be because I didn't use the python script to regrid the emission file (which doesn't work for me). Instead, I used an existing emission file /projects/asci/ancil/ukca_emiss_Monoterp.nc and replaced the name and stashcode with the self-defined tracer name 'ptracer' which gives /projects/asci/ancil/ukca_emiss_ptracerfromM.nc.

Could it be due to the emission file or something else?

Thanks,
Xuemei

comment:8 Changed 10 months ago by luke

Hi Xuemei,

Have you managed to run your suite with the new tracer turned on but without any emissions? You should be able to do this, and the error you're getting is associated with not adding a new tracer correctly rather than anything emissions related.

Can you look towards the end of the job.out file and see if you get any additional output there. Also check that your capitalisation and whitespace for your new tracer are consistent in the nm_spec array and in the chch_defs specification, as well as checking that you have edited the STASHmaster_A file correctly and have initialised the new tracer in Rose. Remember that sometimes you need to close your Rose suite GUI and make changes to the rose-*.conf files before opening the suite again. You should frequently fcm commit your suite changes too, as this helps diagnose if settings have been overwritten. Also, fcm commit your branch.

Once the new tracer exists try the emissions file again.

Essentially, follow steps in the UKCA tutorials for 4 & 5, but for your tracer and not ALICE and BOB:

Thanks,
Luke

comment:9 Changed 10 months ago by xuemeiwang

Hi Luke,

Thanks!

I did run the model with only tracer switched on in section 34 and it worked for me. But when I added the emission, this tracer is then moved to section 50.

The end of job.out file has a table of 'UKCA Prognostics and Diagnostics from D1' printed. Before that, I see 'eg_correct_tracers species / mass conservation error = 6 0.000000000E+00'. I'm not sure if this is related to the error.

nm_spec: I did put a ptracer at slot 173 in this module. Now it's removed. As the stashcode for ptracer is in section 50 but there doesn't seem to be a place to put the tracer from section 50.

Thanks,
Xuemei

comment:10 Changed 10 months ago by luke

Hi Xuemei,

You shouldn't move the tracer out of section 34 - this is the prognostic UKCA section containing the tracers. Section 50 only contains diagnostics. When you add the emissions you don't change anything about the tracer itself, although you do need to add a diagnostic for any emissions into that tracer (which will be in section 50).

Essentially, go back to your s34 changes which worked and test that they still do, and then add the emissions changes on top. Don't change anything about the tracer when adding emissions. Removing things from s34 is the problem.

That error/message likely has nothing to do with this problem.

Thanks,
Luke

Thanks,
Luke

comment:11 Changed 5 months ago by ros

  • Resolution set to answered
  • Status changed from new to closed

Closed due to lack of activity

Note: See TracTickets for help on using tickets.