Opened 14 months ago

Closed 8 months ago

#2577 closed help (fixed)

Switching on monthly updating of vegetation fraction ancillary

Reported by: charlie Owned by: um_support
Component: UM Model Keywords: ancillary file
Cc: Platform: ARCHER
UM Version: 10.7

Description

Hi,

I was very much hoping you would be able to provide some advice on how to resolve an error I am getting when using a new vegetation fraction ancillary file. Please note that, although I have been asking lots of questions about this file in other tickets, this is completely unrelated to other tickets e.g. #2558.

In short, what I'm trying to do is get a monthly varying inland water fraction - which therefore also means changing the fractions of the other land types too. I am primarily interested in the lake fraction, and the plan is to run the model but with this varying throughout the year. So I am wanting to change the lake fraction, tile 7, within this file - we already have a process for creating the new lake fraction, but in the meantime I am using a dummy file just to get things working, created by repeating the existing single timeslice 12 times. The updating needs to be done monthly, i.e. the new lake fraction needs to be updated every month which, by definition, requires the other tiles to be monthly as well.

I have now made my dummy ancillary, with 12 repeating timeslices (to represent 12 months) - this is on Archer, at /home/n02/n02/cjrw09/ancils/tilaka/ancils/qrparm.veg.frac

In my existing suite on Archer (u-ba289), the previous veg fraction is physically in the dump I'm restarting from i.e. it is not listed within um > namelist > Reconfiguration and ancillary > control > Configure ancils. Therefore, I was advised that I needed to create my own section for this, in order to switch on updating. Therefore, I firstly changed my ancillary versions file (in Rose at suite.conf > install_ancil > file) so that it points to my new ancillary, as above, and then I created a new ancillary section using the following settings:

stash_req = 0: Primary fields, FRACTIONS OF SURFACE TYPES (216)
ancilfilename = /home/n02/n02/cjrw09/ancils/tilaka/ancils/qrparm.veg.frac
domain = whole grid
update_anc = true
period = months
interval = 1
source = initialise from ancillary

However, and as I feared, it failed more or less straight away, at the recon stage (although at least it got past the install_ancil stage). It's given me the error below, which is obviously something to do with the updating but I don't understand what:

???????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 816
?  Error from routine: Rcf_Ancil_Atmos
?  Error message: replanca_rcf_replanca: Non-standard period for periodic data
?  Error from processor: 16
?  Error number: 1
????????????????????????????????????????????????????????????????????????????????

I thought to begin with that the problem might be to do with Rose interpreting "monthly" incorrectly, so instead I tried changing it to 30 day updating instead. It failed at exactly the same stage and with exactly the same error. Another idea I had was that the ancillary might not be using the correct calendar, but it does indeed use a 360 day calendar which is what Rose is expecting. Another possible reason I wondered about is that my times (from my ancillary) are currently set as:

0001/01/01:00.00 / 0.000000
0001/01/02:00.00 / 1.000000
0001/01/03:00.00 / 2.000000
0001/01/04:00.00 / 3.000000
0001/01/05:00.00 / 4.000000
0001/01/06:00.00 / 5.000000
0001/01/07:00.00 / 6.000000
0001/01/08:00.00 / 7.000000
0001/01/09:00.00 / 8.000000
0001/01/10:00.00 / 9.000000
0001/01/11:00.00 / 10.000000
0001/01/12:00.00 / 11.000000

Might these be incorrect? Obviously I had nothing to compare to when making these times, because the original ancillary only has one timeslice. If this is the problem, how can I resolve it?

Any ideas you have would be greatly appreciated, as always.

Charlie

Change History (22)

comment:1 Changed 14 months ago by simon

Hi,

In the first instance, change the "initialise from ancillary" to "set to missing data". As the ancil is time varying it is updated as the model runs and shouldn't be going through the recon. However, it's likely that you'll get the same error in the model. Looking at the times above and at the headers in your ancil file, and it appears that the ancil has a periodicity of 12 days rather than a year.

Simon.

comment:2 Changed 14 months ago by charlie

Hi Simon,

As I feared, the model failed again after making your suggested change above. However, it's given me a different although similar error this time, and it failed slightly later (at the coupled stage, rather than recon, which succeeded this time). Error is below:

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 602
?  Error from routine: UP_ANCIL
?  Error message: REPLANCA: Non-standard period for periodic data
?  Error from processor: 612
?  Error number: 11
????????????????????????????????????????????????????????????????????????????????

Is this relating to the same problem, just later on down the line because it is no longer being dealt with by recon?

Charlie

comment:3 Changed 14 months ago by simon

Hi,

Have you reprocessed the ancil? I should have been more explicit, you need to regenerate the ancil with the correct dates. The fields are currently have validity dates for 1st-12th Jan, ie the interval is 1 day. You need to regenerate to change this interval to a month.

comment:4 Changed 14 months ago by charlie

Okay Simon, I will try that. What exact format should the times take? At the moment, I am creating a file with:

0001/01/16:00.00 / 15.000000
0001/02/16:00.00 / 45.000000

and so on. Is that correct?

comment:5 Changed 14 months ago by charlie

Alternatively, I could create a file with:

0001/01/01:00.00 / 0.000000
0001/02/01:00.00 / 30.000000
0001/03/01:00.00 / 60.000000

and so on. Which is correct?

comment:6 Changed 14 months ago by simon

It depends on the input data. If they are monthly means over a calendar month, then the first one, ie the one with the 16th of the month as the validity time. This indicates that the values come from a calendar monthly mean centred at 0000Z on the 16th.

comment:7 Changed 14 months ago by charlie

Yes, they are monthly means over a calendar month, so I will go with the first one and let you know what happens…

comment:8 Changed 14 months ago by charlie

Hi Simon,

Good news, it appears to be running! At least it started the coupled stage almost 10 minutes ago, and appears to be going. I am planning to run it for a year, with 6 month cycling, so will let you know if it fails before it gets to the end. Fingers crossed though.

Many thanks,

Charlie

comment:9 Changed 14 months ago by charlie

Hi Simon,

Further to my last message, it appears to have worked. At least it ran for the full year, both cycles, and produced output. Is there anyway I can check that it did indeed read in my new monthly vegetation fraction ancillary, rather than reverting back to whatever it was using before (presumably included in the dump)? I have checked my restart dump from my output, and the fractions field only has one time slice - given that the input file had 12, does this imply it hasn't worked?

Charlie

comment:10 Changed 14 months ago by simon

Hi Charlie,

The dump only holds the required time slice, taken (and interpolated if required) from the ancil.
There are a number of ways to check.
Firstly, if you have archived monthly dumps, look at the veg frac in these.
Secondly, look at the output from the run, search for lines like:

REPLANCA: UPDATE REQUIRED FOR FIELD 2
Information used in checking ancillary data set: position of lookup table in dataset: 10
Position of first lookup table referring to data type 1
Interval between lookup tables referring to data type 9 Number of steps 1
STASH code in dataset 216 STASH code requested 216
'start' position of lookup tables for dataset 2 in overall lookup array 1021

There will be a number of groups of these, but you're interested in the group with the line with STASH code 216, which is for veg frac. This shows the model is reading in the correct ancil. This group should be repeated each time you update the ancil. So (if you are doing monthly updating) the "Information used in checking ancillary data set: position of lookup table in dataset:" value should increase by 9 each at the start of each month (as there are 9 fields in the ancil).

comment:11 Changed 14 months ago by charlie

Thanks Simon. Which file would this output be in, and where?

comment:12 Changed 14 months ago by charlie

Further to my last message: I have found it I think (job.out for a given cycle) and have found the relevant section, but there doesn't appear to be more than one "field 2" and I can't seem to see it repeating every month. Am I doing something silly?

Last edited 14 months ago by charlie (previous) (diff)

comment:13 Changed 14 months ago by simon

Did you look through the entire file? Do a search for " 216 ". That's two spaces, 216, then 2 spaces.
If the job is ba289. I've had a look at the output files on Archer (there's two 6 month files under /home/n02/n02/cjrw09/cylc-run/u-ba289/log/job/) and the output is looks OK.

comment:14 Changed 14 months ago by charlie

Ok, I think I have found it and I think it all looks okay and it has worked. Given that my cycle contains 6 months, the following should be repeated 6 times, and indeed it is:

REPLANCA: UPDATE REQUIRED FOR FIELD 2
Information used in checking ancillary data set: position of lookup table in dataset: 55
Position of first lookup table referring to data type 1
Interval between lookup tables referring to data type 9 Number of steps 6
STASH code in dataset 216 STASH code requested 216
'start' position of lookup tables for dataset 2 in overall lookup array 1021

Here, everything remains the same, except "Information used in checking ancillary data set: position of lookup table in dataset: 55" which increases by 9 each time i.e. 55, 64, 73, 82, 91, 100 and "Number of steps 6" which increases by 1 each time. Is this correct? If so, very many thanks!

Charlie

comment:15 Changed 14 months ago by simon

  • Resolution set to fixed
  • Status changed from new to closed

That is correct. I'll close the ticket.

comment:16 Changed 11 months ago by charlie

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:17 Changed 11 months ago by charlie

Hi,

I have reopened this ticket, rather than starting a new one, because of a very similar and highly related issue (and indeed the same suite, u-ba289). You would rather I start a new ticket, then let me know.

The above instructions worked absolutely fine for my test a couple of months ago, but I am now trying to do exactly the same thing but with a real vegetation ancillary file that we have created. I'm fairly sure I have created it correctly, following all the instructions above, and although I didn't receive any of the above errors, I have now received a new error (one minute into the coupled stage):

????????????????????????????????????????????????????????????????????????????????
Rank 458 [Tue Nov 27 04:51:00 2018] [c5-0c2s7n3] application called MPI_Abort(MPI_COMM_WORLD, 9) - process 458
???!!!???!!!???!!!???!!!???!!!       ERROR        ???!!!???!!!???!!!???!!!???!!!
?  Error code: 1
?  Error from routine: EG_BICGSTAB
?  Error message: Convergence failure in BiCGstab, omg is NaN
?        This is a common point for the model to fail if it
?        has ingested or developed NaNs or infinities
?        elsewhere in the code.
?        See the following URL for more information:
?        https://code.metoffice.gov.uk/trac/um/wiki/KnownUMFailurePoints
?  Error from processor: 461
?  Error number: 12
????????????????????????????????????????????????????????????????????????????????

Experience tells me that this means there is missing data somewhere, but the only ancillary file I have changed is the vegetation fraction as above. All I have done is change some values, so any missing data should be exactly the same as the one that worked, above. I haven't modified the land sea mask or anything like that.

As per the usual instructions, I am now running the suite again, but with the extra diagnostics turned on in PRINT_STATUS. Can you possibly advise?

Many thanks,

Charlie

comment:18 Changed 11 months ago by simon

Hi Charlie,

It would be useful if you could let us know the paths to the old and new ancils, so that we can compare and contrast.

Simon.

comment:19 Changed 11 months ago by charlie

No problem.

My current ancillary file is at /home/n02/n02/cjrw09/ancils/tilaka/ancils/qrparm.veg.frac

This should be an exact copy of qrparm.veg.frac_martin in the same directory (and is the UM format version of qrparm.veg.frac_martin.nc, also in the same directory).

The old version is qrparm.veg.frac_12mo_new (the UM format version of qrparm.veg.frac_12mo_new.nc), also in the same directory.

There is only one simple difference between the new and the old versions. Both contain vegetation fraction, both with 9 tiles, both repeated over 12 months. The old version is a test, and is simply the original data (which is once timeslice) repeated 12 times. The new version is exactly the same, but tile 7 (ice) being modified by us. Nothing else has changed to any of the other tiles, and tile 7 should be the same each and every month.

Charlie

comment:20 Changed 11 months ago by simon

I've had a look at the ancils. A couple of immediate comments: Tile 7 looks more like a water field, tile 9 appears to be the ice field. Also, did you ensure that all tiles sum to 1.0 for each point?

comment:21 Changed 11 months ago by charlie

Sorry, you are absolutely right, I was having a momentary senior moment. That's the result of modifying multiple things for multiple purposes. You are absolutely right: tile 7 is the water field (or rather lake fraction) which is the one we have modified. We haven't done anything to tile 9 (ice), so it should be identical to the original but just repeated 12 times. I didn't make the file myself so I haven't checked that the tiles sum to 1.0, but I will check right now…

comment:22 Changed 8 months ago by willie

  • Keywords ancillary file added
  • Resolution set to fixed
  • Status changed from reopened to closed
Note: See TracTickets for help on using tickets.