#2546 closed help (fixed)

Turning off aerosols

Reported by: charlie Owned by: um_support
Component: UM Model Keywords:
Cc: Platform: NEXCS
UM Version: 10.7

Description

Hi there,

This question relates to several other tickets (especially 2464, comments 70 onwards) and picks up on previous conversations with Grenville, Ros and Luke, but I have started a new ticket because they were all becoming unwieldy and, upon your suggestion, I have gone back to basics and have started addressing each problem in turn, rather than changing lots of things at once.

So… I am trying to run u-az608. This is an identical copy of u-aw739 (which runs fine, albeit only for 20 years). If I simply run u-az608 without changing anything, it works fine. However, when I then modify the aerosols (listed in /roses/u-az608/app/um/rose-app.conf under run_ukca) such that I remove all but 3 of them, it fails. The aerosols I am keeping are as follows:

$UM_NETCDF_UKCAEMISS_DMS_DIR/$UM_NETCDF_UKCAEMISS_DMS_FILE
$UM_NETCDF_UKCAEMISS_MONOTP_DIR/$UM_NETCDF_UKCAEMISS_MONOTP_FILE
$UM_NETCDF_UKCAEMISS_SO2NAT_DIR/$UM_NETCDF_UKCAEMISS_SO2NAT_FILE

Because I have been told that these are the only natural aerosols whereas the others can all be considered anthropogenic.

I have also been told that because the model ingests multiple ancillary aerosol emissions, and internally combines them into one, it doesn't matter how many there are (as long as there is more than one). So I was told that simply removing the unwanted ones *should* work. However, it clearly doesn't, because in my job.err it is giving me the following error:

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 1
?  Error from routine: UKCA_EMISS_INIT
?  Error message: BC_fossil missing in the emission files
?  Error from processor: 315
?  Error number: 18
????????????????????????????????????????????????????????????????????????????????

Or is this a red herring? Assuming not, this error is accurate, because I have indeed removed this file. However, that's because I want to run without it. How can I force the model to run with only the above 3 aerosols, and not require the others? Is there a way of controlling which ones it uses and which ones it ignores?

Thanks,

Charlie

Change History (52)

comment:1 Changed 16 months ago by luke

Hi Charlie,

What you're doing here is removing emissions files of a complete type. If you had two (or more) files for each type, some of which were anthropogenic, you could remove the anthropogenic file and the model would still run, just without those emissions. However, when the files are defined as, e.g. $UM_NETCDF_UKCAEMISS_DMS_DIR/$UM_NETCDF_UKCAEMISS_DMS_FILE there is only a single file and so it can't be removed.

Also, these aren't aerosols. These are emissions into the aerosol modes, or into chemical tracers that then oxidise to go onto form aerosols. Changing the aerosol configuration is not for the faint of heart and I would not advise this. If you want to remove anthropogenic contributions to aerosols you need to run with pre-Industrial emissions (or another emissions set that you have devised).

I would advise contacting the Met Office and asking about the CMIP6 pre-industrial emissions and see if you can use these. If they haven't already been copied over to Monsoon2 and ARCHER they probably will be soon.

If you have a file that is only anthropogenic the simplest thing to do is make a copy and fill it with zeros. The code just checks to see if the file exists and can be read, not whether there are any non-zero emissions in the file. You don't need to specify the environment variables for these new files, you can just put the full path in.

Thanks,
Luke

comment:2 Changed 16 months ago by charlie

Hi Luke,

Very many thanks. I think I understand - just to clarify, you are saying that if a given netcdf file contains several fields, then one or more of these fields can be removed no problem, but you can't remove the whole file itself. Is that right?

In other words, if I want to just keep the natural emissions and remove the anthropogenic ones, I need to modify the latter and fill them with zeros, so that the file still exists but is just zero?

I have already found the CMIP6 preindustrial anthropogenic emissions on Monsoon, which appear to be at /projects/ancils/cmip6/ancils/n96e/timeslice_1850/AerosolChemistryEmissions/v1 and are as follows:

BC_biofuel_1850_time_slice.nc       OC_biomass_high_1850_time_slice.nc
BC_biomass_high_1850_time_slice.nc  OC_biomass_low_1850_time_slice.nc
BC_biomass_low_1850_time_slice.nc   OC_fossil_1850_time_slice.nc
BC_fossil_1850_time_slice.nc        SO2_high_1850_time_slice.nc
OC_biofuel_1850_time_slice.nc       SO2_low_1850_time_slice.nc

Are all of these essential to make the model run, or can some be ignored?

Interestingly, though, that don't appear to be any CMIP6 preindustrial natural emissions - for the 3 natural emissions (i.e. DMS, Monoterp & SO2_nat), these only appear to exist as CMIP5 year 2000 versions, which are at

/projects/um1/ancil/atmos/n96e/ukca_emiss/cmip5/2000/v2
ukca_emiss_Monoterp.nc
ukca_emiss_DMS.nc

and

/projects/um1/ancil/atmos/n96e/ukca_emiss/aerocom/v1
ukca_emiss_SO2_nat.nc

These are what is currently being used in my Holocene simulations, which are identical to the CMIP6 preindustrial control simulation.

For these 3 natural emissions, my plan was to 'palaeotise' them, which is the term we have invented to simply mean applying a zonal mean to the data and using this instead.

So, if I understand you correctly, for the model to run I need to include all of the above emissions, both anthropogenic and natural. For the 3 natural ones, I need to 'palaeotise' them. Conversely for the others, I need to keep them present but fill them with zeros.

Is that all correct?

Charlie

comment:3 Changed 16 months ago by luke

Hi Charlie,

No - I'm not quite saying this. Sorry for not being clear. The convention is that each NetCDF file contains a single field, but this isn't necessarily the case. You could conceivably have a single NetCDF file with all fields in, although if its like this then you'd only have a single field for each emitted species.

You should include all the CMIP6 emissions you list above, although you may want to change the values (see below). If you look at the filenames these are all for separate things. If you opened them up and viewed the variable names you would see that these are all different. UKCA is expecting at least ONE emissions variable defined for each species. There can be more than one, and these will be in separate files, often grouped by anthropogenic/natural etc. If this is the case then you can just remove the anthropogenic file, leaving the natural one, but this is NOT the case here.

I'm not sure what is planned for the natural emissions. Usually these are a single-time or annually-repeating climatology, rather than a time-series. It is probably fine to just use the CMIP5 ones. If you want to modify them, that's fine too. You will need to include DMS, Monoterpene and SO2 natural emissions fields via NetCDF in some form.

I'm probably confusing matters more here. It shouldn't be too difficult:

  1. In the code there is a list of species that are emitted into. You should not change this (unless you really know what you are doing, and then at your own risk).
  2. Because of (1) you need to provide an emissions field for each species you emit into, although you can provide multiple fields and these are summed by the code.
  3. All emissions files you specify must provide all fields, but only after they have all been read-in.

Note that there is NOT and 1:1 relationship between emissions files (read-in by the model) and emissions fields (added to tracers within UKCA).

Have a look at each of the CMIP6 files - it might be that these are zeros anyway. If not, then you will want to zero them. However, you must still include them in some form. If you look at the variable names these specify what they are.

For more information, take a look at the UKCA emissions tutorial:

http://www.ukca.ac.uk/wiki/index.php/UKCA_Chemistry_and_Aerosol_vn10.9_Tutorial_5

These are for chemical emissions, but the principles are similar.

I hope this helps.

Thanks,
Luke

comment:4 Changed 16 months ago by charlie

Thanks Luke, and sorry for the delay, it's been a busy morning of meetings. I think I understand.

If the first instance, I'm going to try running my suite with the CMIP6 preindustrial versions (anthropogenic) and CMIP5 modern versions (natural), completely unmodified and including all, as above. This really should work, because these are currently being used in my Holocene simulations which work fine.

Assuming this works, I will then try modifying them according to what we discussed above i.e. "palaotising" the 3 natural emissions and filling the others with zeros.

I'll let you know what happens…

Charlie

PS. When it comes to creating new files of the above and filling them with zeros, is there easier way of doing this - at the moment I am using IDL to create my new ancillary files, but the downside of this is that the metadata needs to be put into the new file manually, which is a chore. Is there an easier way of just taking an existing file, overwriting the data with zeros, and writing out a new version with the same metadata, dimensions, etc?

comment:5 Changed 16 months ago by luke

Hi Charlie,

If the first instance, I'm going to try running my suite with the CMIP6 preindustrial versions (anthropogenic) and CMIP5 modern versions (natural), completely unmodified and including all, as above.

I agree that this is the best plan. I always advise "baby steps" where possible to enable you to identify which step is causing the problem.

I would suggest python. I know that the standard NetCDF4 library should allow you to read-in the file, zero what-ever data fields you like, and then save them again in essentially the same format. I suspect that Iris would allow you to do the same too.

There is also an example (making a new file, but this can be modified) using Iris on the UKCA tutorial I link to above.

Thanks,
Luke

comment:6 Changed 16 months ago by charlie

Hi Luke,

Right then… The good news is that when I ran my suite with the existing unmodified 13 emissions files (10 CMIP6 preindustrial anthropogenic + 3 CMIP5 modern natural), it worked fine.

However, when I then modified all of these (by filling the 10 anthropogenic files with zeros and the 3 natural files with zonal means) and ran again, it failed at the first timestep of the atmos_main stage (it created the first restart dump but then failed). I'm assuming (perhaps incorrectly) that it's not a problem with the metadata of my modified files (e.g. schoolboy errors like being upside down or the dates are incorrect), because firstly the recon stage succeeded and secondly I have double- and triplechecked the files and they are identical in every way to the originals (apart from their actual data, of course).

If I look at my error log, I can see the following error but I don't know what this means?:

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 65536
? Error from routine: EM_GET_TIME_INFO
? Error from processor: 0
? Error number: 18
????????????????????????????????????????????????????????????????????????????????

Obviously the fact that I have changed all the files at once complicated things, but I was being optimistically hopeful. Do the error logs show which (or several) of the 13 files is causing the problem? If not, then the only way I can see of finding out which one is causing the problem is to add in the new files one by one, resubmitting each time, until it fails. Do you have any other suggestions?

Thanks,

Charlie

comment:7 Changed 16 months ago by luke

Hi Charlie,

From the routine giving the error it seems like something has messed-up with the dates.

The error message you've given doesn't help track it down though, as the Error code is likely from the NetCDF routines and so isn't very clear. However, in the job.out file there should be a message printed that gives more information. Are you able to find this and post it here?

Many thanks,
Luke

comment:8 Changed 16 months ago by charlie

The only error I can see the job.out is as follows:

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 65536
? Error from routine: EM_GET_TIME_INFO
? Error message: Time coordinate: time has unrecognised calendar: 360_day
????????????????????????????????????????????????????????????????????????????????

So I guess this is implying that there is indeed a problem with the dates. But it still doesn't say where?

Charlie

comment:9 Changed 16 months ago by luke

Sorry for not being clear. There should be a message printed before this error message is - most likely in the job.out file. It will not be within the ? block, but separate to it.

Thanks,
Luke

comment:10 Changed 16 months ago by charlie

Somewhere here?

UKCA AGE-OF-AIR: Reset method= 1. Tracer will be reset upto level 10
5 files found in offline namelist
O3 found in projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_O3.nc
OH found in
projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_OH.nc
NO3 found in projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_NO3.nc
H2O2 found in
projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_H2O2.nc
HO2 found in projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_HO2.nc
Number of NetCDF offline fields: 5
UKCA_OFFLINE_OXIDANTS_UPDATE: Update first data field for target time 1988 9 3 12 0 0
GET_EMFILE_REC: Get data field for 1988 9 3 12 0 0 602 43200 14460.000
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
GET_EMFILE_REC: Time found between these regs. in NetCDF file: 8 9 14040.000 14760.000
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: O3
Oxidant level 1: (max, min, mean) 0.300E-07 0.288E-07 0.293E-07
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: OH
Oxidant level 1: (max, min, mean) 0.988E-16 0.445E-16 0.638E-16
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: NO3
Oxidant level 1: (max, min, mean) 0.475E-13 0.358E-13 0.420E-13
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: H2O2
Oxidant level 1: (max, min, mean) 0.137E-10 0.929E-11 0.117E-10
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: HO2
Oxidant level 1: (max, min, mean) 0.123E-12 0.651E-13 0.862E-13
13 files found in emission namelist

comment:11 Changed 16 months ago by charlie

Further to my last message, I think I have found the problem. There is indeed a mismatch between my dates and the original dates in some of the files. I will correct, and get back to you…

comment:12 Changed 16 months ago by charlie

Okay I have now corrected the error, meaning the dates are all identical to the original versions. However, upon resubmitting, I get exactly the same error in both the job.err and job.out:

UKCA AGE-OF-AIR: Reset method= 1. Tracer will be reset upto level 10
5 files found in offline namelist
O3 found in projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_O3.nc
OH found in
projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_OH.nc
NO3 found in projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_NO3.nc
H2O2 found in
projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_H2O2.nc
HO2 found in projects/um1/ancil/atmos/n96e/oxidants/ccmi_refc1_anqdg/clim_1988_2010/v1/ukca_oxid_clim_HO2.nc
Number of NetCDF offline fields: 5
UKCA_OFFLINE_OXIDANTS_UPDATE: Update first data field for target time 1988 9 3 12 0 0
GET_EMFILE_REC: Get data field for 1988 9 3 12 0 0 602 43200 14460.000
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
GET_EMFILE_REC: Time found between these regs. in NetCDF file: 8 9 14040.000 14760.000
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: O3
Oxidant level 1: (max, min, mean) 0.300E-07 0.288E-07 0.293E-07
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: OH
Oxidant level 1: (max, min, mean) 0.988E-16 0.445E-16 0.638E-16
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: NO3
Oxidant level 1: (max, min, mean) 0.475E-13 0.358E-13 0.420E-13
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: H2O2
Oxidant level 1: (max, min, mean) 0.137E-10 0.929E-11 0.117E-10
EM_GET_TIME_REC: File starts 1970 1 1 0 0 0 709200 0
Updated Oxidant at time: 1988 9 1 0 20 0 14460.00000
Oxidant name: HO2
Oxidant level 1: (max, min, mean) 0.123E-12 0.651E-13 0.862E-13
13 files found in emission namelist

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 65536
? Error from routine: EM_GET_TIME_INFO
? Error from processor: 0
? Error number: 18
????????????????????????????????????????????????????????????????????????????????

comment:13 Changed 16 months ago by luke

When you resubmitted your job, did you change the names of the files when you corrected the dates, and update your rose-app.conf file accordingly? If so, did you do a rose suite-run --reload?

Could you up the debug level to get more print information coming out?

comment:14 Changed 16 months ago by charlie

No, I didn't change the file names. I simply deleted the old files, and then copied over the new ones (with exactly the same names). The files listed within rose-app.conf should therefore be correct, because the names haven't change. I then did rose suite-shutdown followed by rose suite-run . Is this not correct? How do I up the debug level?

comment:15 Changed 16 months ago by luke

Can you search for PrStatus (or PRINT_STATUS) in your suite, and up it by one level? It might get more information out as to exactly where you are. If it goes too high you get a lot out, so you will need to remember to turn it down once the suite it working.

Last edited 16 months ago by luke (previous) (diff)

comment:16 Changed 16 months ago by charlie

I have just checked, and PRINT_STATUS is already set to its highest level, "Extra diagnostic messages".

comment:17 Changed 16 months ago by luke

Which machine is this being run on - is it possible for me to see the complete job.out and job.err files?

comment:18 Changed 16 months ago by charlie

Sorry for the delay, and yes of course. My suite (u-az608) on NEXCS, specifically exvmsrose. You can view my output logs here: ~/cylc-run/u-az608/log/job/19880901T0000Z/atmos_main/NN.

Many thanks,

Charlie

comment:19 Changed 16 months ago by luke

Hi Charlie,

Can you give me your Monsoon2 username? The ~ will expand to my home directory, not yours. If you could do a pwd in your home directory and tell me the output from that, that might be best.

Thanks,
Luke

comment:20 Changed 16 months ago by charlie

Sorry, yes of course. My username is cwilliams, and my home directory is /home/d05/cwilliams

comment:21 Changed 16 months ago by luke

Hi Charlie,

If you look at the error message now, it's changed. It now reads:

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 65536
?  Error from routine: EM_GET_TIME_INFO
?  Error message: Time coordinate: time has unrecognised calendar: 360_day^@
?  Error from processor: 0
?  Error number: 18
????????????????????????????????????????????????????????????????????????????????

I think that the key thing here is the 360_day^@, i.e. there is a strange character at the end of the calendar, probably from something like a tab or carriage return introduced in a (likely) windows-based text editor. Can you remove this character and re-run? You might need to check all the files, as this error will have occurred on the first file being read, so there may be others.

Thanks,
Luke

comment:22 Changed 16 months ago by charlie

Sorry Luke, I don't entirely understand. The files are all netcdf, so where do I need to check the text? I have just double checked the metadata for each of the files (which I usually write out to a text file using ncdump), and the calendar is the same for each:

double time(time) ;

time:axis = "T" ;
time:bounds = "time_bnds" ;
time:units = "days since 1850-01-01 00:00:00" ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:calendar = "360_day" ;
time:calendar_flexible = "1" ;

As you can see, there is no funny character after 360_day - I have checked all of the files, and this is the same in each. The only thing that varies in some of the files is the days since…

Charlie

comment:23 Changed 16 months ago by luke

This is a bit of a horrible and insidious error in that it's hard to spot. Using ncdump isn't showing it. When you made your new files, how did you do this? Is there a line somewhere which says to set the calendar attribute to 360_day? If so, take a look a this line, ideally in an editor like vim.

comment:24 Changed 16 months ago by charlie

I am using IDL to make these files - I know you said to use python, but personally I can't stand it and don't know it very well. I have always used IDL to create files and never had a problem in the past. I have just checked my code for all of my emissions, and there is no funny character after any of the 360_day - all on my code looks like the following (again, the only thing that varies is the "days since…". Some of the files have the calendar_flexible, some don't. Also, some of them have realtopology as 'linear' whereas others ignore this - but either way, the time attributes for every one of my files matches the original).

NCDF_ATTPUT, fid, tvid, 'axis', 'T'
NCDF_ATTPUT, fid, tvid, 'bounds', 'time_bnds'
NCDF_ATTPUT, fid, tvid, 'units', 'days since 1750-01-01 00:00:00'
NCDF_ATTPUT, fid, tvid, 'standard_name', 'time'
NCDF_ATTPUT, fid, tvid, 'long_name', 'time'
NCDF_ATTPUT, fid, tvid, 'calendar', '360_day'
NCDF_ATTPUT, fid, tvid, 'calendar_flexible', '1'

comment:25 Changed 16 months ago by charlie

Further to my last message, this is very puzzling (and annoying!). I have looked at the data in several ways, and nowhere is there any funny character after 360_day - not in any of the code used to create the files, not in any of the metadata as shown by ncdump, and not even when I view the data files themselves using xconv - here, all of the files say that the correct "360-day calendar is being used". So it is obviously recognising the attribute from my code, because if it wasn't it wouldn't recognise it here. So why is the model seeing another character?

comment:26 Changed 16 months ago by luke

From the text above it seems OK - but this character could be hidden. Of course, it might always be something else as well.

While I appreciate you don't like python, but the reason I was suggesting to use it was because it is possible to essentially edit the file itself rather than making a new file - see below. This then means that you shouldn't have problems like this as the settings already work.

Here is how you could use python to do this. Unfortunately due to the format of the original files, there is an extra step:

  1. Convert the format of the file to NETCDF4-CLASSIC so python can read/write it correctly (otherwise you get a HDF error):
    nccopy -k 4 /projects/ancils/cmip6/ancils/n96e/timeslice_1850/AerosolChemistryEmissions/v1/BC_biofuel_1850_time_slice.nc BC_biofuel_1850_time_slice.nc 
    
  1. Use the following python commands to clear the data to zero:
    import netCDF4
    emiss=netCDF4.Dataset('./BC_biofuel_1850_time_slice.nc','r+')
    emiss.variables['emissions_BC_biofuel'][...]=0.0
    emiss.sync()
    emiss.close()
    

This will now leave you with a new file with the same name (although in NetCDF4-classic format) with all zeros in the data variable. You can use the python command

emiss.variables

to find the correct name for the emissions field - for this file it is:

In [3]: emiss.variables
Out[3]: OrderedDict([(u'emissions_BC_biofuel', <netCDF4.Variable object at 0x39af3d0>),
(u'latitude_longitude', <netCDF4.Variable object at 0x39af450>),
(u'time', <netCDF4.Variable object at 0x39af4d0>),
(u'time_bnds', <netCDF4.Variable object at 0x39af550>),
(u'model_level_number', <netCDF4.Variable object at 0x39af5d0>),
(u'latitude', <netCDF4.Variable object at 0x39af650>),
(u'latitude_bnds', <netCDF4.Variable object at 0x39af6d0>),
(u'longitude', <netCDF4.Variable object at 0x39af750>),
(u'longitude_bnds', <netCDF4.Variable object at 0x39af7d0>)])

You should be able to use this new file in the model without problems. Note that if you use Xconv, you should use the version in /projects/um1/bin and not the version in /usr/local/bin on the postproc to view this file as the version there has an issues with NetCDF4 files.

comment:27 Changed 16 months ago by luke

I should add that the above should all work on the Monsoon2 postproc - they did for me.

comment:28 Changed 16 months ago by charlie

Okay, it looks like I need to be dragged kicking and screaming into the world of python! I will give this a go on the first file and check it works, if so I will then repeat on the others…

comment:29 Changed 16 months ago by luke

As an FYI - this is the command I use to run python:

ipython --pylab --logfile=ipython-`date +"%Y%m%d-%H%M%S"`.py

The logfile bit is just so it saves the commands you run to see later.

Last edited 16 months ago by luke (previous) (diff)

comment:30 Changed 16 months ago by charlie

Okay, thanks. Just a very quick, and basic question: presumably all of the above, including converting the file, needs to be done within python? If so, is it best to write a python file (e.g. test1.py) and then run this, or is it best to run the above lines interactively within python? Either way, how do I begin python on Nexus, as I've never used it on this platform before?

comment:31 Changed 16 months ago by luke

Step (1) should be done on the command-line, step 2 can be done with a python script. You could combine them all together in single script that copies and coverts, and then uses python to zero things.

comment:32 Changed 16 months ago by luke

Also, to use python just ssh into the postproc and use python2.7, or the ipython command I mention above.

comment:33 Changed 16 months ago by charlie

Great, that seems to work on the first file. I'm now in the process of combining it all into a script and modifying all the files.

However, although this should be fine for the files I am zeroing, it has occurred to me that this won't solve the problem for the other 3 natural files i.e. the ones to which I am applying a zonal mean. Shall I see if the model runs anyway, with these new zeroed files?

comment:34 Changed 16 months ago by luke

As always - "baby steps". Make sure the model runs with the new zeroed files and the original natural emissions. Then work on the zonal-meaning.

You can always do this in python anyway, e.g.

import netCDF4
emiss=netCDF4.Dataset('./foo.nc','r+')
zm_emiss=netCDF4.Dataset('./bar.nc','r')
emiss.variables['emissions_NAME'][...]=zm_emiss.variables['emissions_NAME'][...]
emiss.sync()
emiss.close()

where foo.nc is one of the original natural files, and bar.nc is one you've made with the zonal-meaning applied. You could always do the zonal-meaning in python as well if you wanted.

The use of python is just to ensure that you're not changing the metadata.

comment:35 follow-up: Changed 16 months ago by charlie

Fair enough, baby steps! Okay, I will try running the model with the zeroed anthropogenic files (as created by python) and the original natural ones, and let you know what happens.

But, when it comes to it, are you saying I can still use my existing IDL code to create the zonal meaning, and just write this out as a netcdf but without worrying about any attributes, then use the python code above to read in the original version (foo.nc) and my modified one (bar.nc), resulting in a new file being created which has all the data from bar.nc but the attributes from foo.nc? If so, that would be great and would save me a lot of bother, both now and in the future. I asked someone a long time ago if there was a way of reading in attributes from a file and automatically applying these to a new file, but was told they didn't know of any - so I have been doing it manually, which is not only very laborious but is also open to errors (as we may be seeing here).

comment:36 in reply to: ↑ 35 Changed 16 months ago by luke

But, when it comes to it, are you saying I can still use my existing IDL code to create the zonal meaning, and just write this out as a netcdf but without worrying about any attributes, then use the python code above to read in the original version (foo.nc) and my modified one (bar.nc), resulting in a new file being created which has all the data from bar.nc but the attributes from foo.nc? If so, that would be great and would save me a lot of bother, both now and in the future. I asked someone a long time ago if there was a way of reading in attributes from a file and automatically applying these to a new file, but was told they didn't know of any - so I have been doing it manually, which is not only very laborious but is also open to errors (as we may be seeing here).

Yes, this is what I'm saying. Technically, what you're doing isn't reading the attributes from a file and copying them, but instead over-writing the data in the original file (so always work on a copy!), hence keeping all the other attributes the same. If you need to change these you can also do that by editing the other variables. It will get more complicated if the dimensions change though - this only works in the examples I've given if the only thing changing is the values of the data.

comment:37 Changed 16 months ago by charlie

Excellent, understood. As I said above, I'm just about to submit my suite with the 10 zeroed anthropogenic files but original 3 natural ones. I'll let you know what happens…

comment:38 Changed 16 months ago by charlie

Okay, the good news is that it appears to be running so far, i.e. it has run for over 10 minutes so far and has generated 2 restart dumps. Last time, it failed after 3 minutes. Assuming that it's safe, I'm going to try shutting it down and resubmitting with the newly modified (i.e. using the python code) 3 natural files as well…

comment:39 Changed 16 months ago by charlie

Hi Luke,

I got all excited then, because it ran for just over 25 minutes using the new emissions (both the zeroed anthropogenic files and the zonal meaned natural files) but then failed, giving me a brand new error this time:

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 2
? Error from routine: U_MODEL_4A
? Error message: ACUMPS1: Partial sum file inconsistent. See Output
? Error from processor: 533
? Error number: 572
????????????????????????????????????????????????????????????????????????????????

I have no idea what this means but I guess it must be to do with one of the new emissions, either the zeroed ones or the meaned ones. Can you possibly advise? When it says 'See output' what does it mean?

If you want to look at the actual files, they should be in the same place as above.

Thanks,

Charlie

comment:40 Changed 16 months ago by luke

Hi Charlie,

This probably doesn't mean that there are any problems with your new files. Looking at #2397 and #2163 it seems like the problem could be occurring as things have changed from the initial run.

When you swapped your files in, did you do a rose suite-clean or use a new job? How long did your original job run with the "working" emissions (i.e. the original pre-industrial and natural ones)? It may be that there are some partial sum files hanging around from this original run that are causing problems.

I would suggest trying a rose suite-clean and sending the suite off again as a NRUN and seeing if this fixes it.

Thanks,
Luke

comment:41 Changed 16 months ago by charlie

Okay, understood, I think. No, I didn't do a rose suite-clean , in fact I have never heard of this. I simply shut down my previous job, and then ran the NRUN i.e. rose suite-shutdown and then rose suite-run . I thought this meant it started from a fresh, particularly as I still have Build and Run Recon switched on, which I thought meant that it started completely from scratch each time I run?

With the working emissions: I honestly can't remember, sorry. I thought I let it run for about half an hour, but perhaps it was less so perhaps the problem is less to do with my modified files and more to do with these?

Either way, shall I go ahead with rose suite-clean , delete everything which has been run so far using this job, and then start from scratch?

comment:42 Changed 16 months ago by luke

What rose suite-clean will do is delete everything in your cylc-run directory, effectively giving you a pristine environment to work with as if you had never run the job before. It means it will need to recompile things afresh etc. The roses directory will be untouched.

While you shouldn't have had problems not doing this, this is a good thing to check just to make sure.

comment:43 Changed 16 months ago by charlie

Hi Luke,

Right then, I don't want to speak too soon and jinx everything, but it has now been running for over an hour and seems to be stable - this is using all the modified files, made with your python i.e. the 10 anthropogenic files which have been zeroed, and the 3 natural files which have been zonal meaned. It has written out 7 restart dumps and is currently on its 3rd month. I plan to leave it going for the next day, so that it gets through an entire cycle (which should take 20 hours and should produce 4 years by the end), but as a general rule do you think it's safe to assume it's now working?

Charlie

comment:44 Changed 16 months ago by luke

That certainly sounds more promising! I'd let it finish and then take a good look at the results to see what they are like and if anything looks screwy, but from a technical perspective that is good news.

comment:45 Changed 16 months ago by charlie

Excellent, many thanks. Do you have any feeling as to what might look screwy i.e. what I would need to look at?

comment:46 Changed 16 months ago by luke

I'm not sure, sorry. It shouldn't look too different from what you've run previously. If you get a massive amount of aerosol or strange temperature spikes etc. it could indicate issues, especially as the aerosols will affect the radiation. Of course, there should be differences and this is the interesting bit!

comment:47 Changed 16 months ago by charlie

Okay, understood, I will check everything once it has run a full cycle.

In the meantime, onto my next issue which, I'm sure you will be pleased to hear, has nothing to do with aerosols so I shall probably need to bother someone else. I will submit a new ticket, because my next problem is unrelated to this ticket.

Perhaps, though, don't close this ticket just yet, until it has run a full cycle and I have checked everything looks okay. This should be done over the weekend.

Very many thanks for all your help, very much appreciated.

Charlie

comment:48 Changed 16 months ago by luke

Hi Charlie,

No problems - happy to help.

I'm away next week, so please feel free to close this ticket next week if you are satisfied, by picking the appropriate "resolve as" setting under the "Modify Ticket" option.

Thanks,
Luke

comment:49 Changed 16 months ago by charlie

No problem, will do.

One more quick question: will the python code that you sent me, using the emiss functions, do the same thing with any netcdf file e.g. if I give it an original version and a modified version, will it copy the modified data over and thus maintain the attributes? Or, as the function name suggests, does this only work on emissions files?

Charlie

comment:50 Changed 16 months ago by luke

It will work for everything - emiss is just the variable name that I chose to use in python. I could have picked anything, a, f, geoff etc.

comment:51 Changed 16 months ago by charlie

Excellent, many thanks, that's really useful.

comment:52 Changed 16 months ago by charlie

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.