Opened 2 months ago

Closed 4 weeks ago

#2618 closed help (fixed)

Error in CREATEBC causes frames step to fail

Reported by: shakka Owned by: um_support
Priority: normal Component: UM Model
Keywords: createbc, frames Cc:
Platform: Monsoon2 UM Version: 11.1

Description

Hi CMS,

I have encountered the error below in the innermost nest of my nesting suite (glm N768 > 4 km > 1.5 km), which I can only assume is due to the LBCs being created improperly by the larger domain.

????????????????????????????????????????????????????????????????????????????????
???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!
? Error code: 999
? Error from routine: CREATEBC
? Error message: No fields written to disk.
? Error from processor: 0
? Error number: 0
????????????????????????????????????????????????????????????????????????????????

The 4 km nested domain runs fine, but the 1.5 km nest fails on the frames step. Looking at the job.out files, it appears that there is nothing written to any of the umnsaa_cb* files, which produces this error.

I wasn't aware of having changed any settings related to LBC creation, so I am a bit stumped about this error. Could you please advise? The suite ID is u-ba903.

Thanks,
Ella

Change History (12)

comment:1 Changed 2 months ago by willie

Hi Ella,

It fails in task Peninsula_1p5km_nest_test_um_frame_000 with the error you mention. This is CreateBC trying to read the 24 files

share/cylc/<T>/Peninsula_4p0km/nest_test/um/umnsaa_cb*

These exist and have header information, but no data. So the question is why the 4km model is not creating the STASH data required by CreateBC.

The LAM LBC data is defined in um/opt/rose-app-lbc.conf which has not been modified, although I am a bit concerned about the mixing of CB_FREQ and LBC_FIELD_FREQ.

Other changes that have been made include,

  • the addition of the droplet number branch,
  • modifications to um/rose-app.conf (21 changes),
  • modifications to um/opt/rose-app-verstash.conf (156 changes)

I think the most likely source of the problem is in the changes made to um/rose-app.conf.

This is as far as I have got.

regards
Willie

comment:2 Changed 2 months ago by shakka

Hi Willie,

As far as I understand, the changes to rose-app.conf and rose-app-verstash.conf are just the changes to the STASH variables that I want outputted. I'm also not sure I understand what you mean about the mixing of CB_FREQ and LBC_FIELD_FREQ - I've never intentionally changed those.

Cheers,
Ella

comment:3 Changed 2 months ago by willie

Hi Ella,

I think the issue is that the STASH changes in um/rose-app.conf have prevented the 4km model from outputting the STASH it needs to support CreateBC.

I ran the Metadata -> Check All Validator Macros on the STASH Requests page and it came up with one error - STASH 0,217 Cloud Ice is an error. You could try turning this off and running again for one cycle.

Regards
Willie

comment:4 Changed 2 months ago by willie

Hi Ella,

Did you use the copy_stash macro to add STASH? If you did there is a note about it at https://code.metoffice.gov.uk/trac/rmed/wiki/suites/nesting/copystash.

PS: You should repeat the Validator macro after removing 0,271 just in case it is hiding anything and also re-index with TidyStashTransform? just to make sure.

Willie

comment:5 Changed 2 months ago by shakka

Hi Willie,

I tried your suggestion, but unfortunately it didn't work. I'm still getting the same error. The 4 km nest isn't outputting anything, despite the rg01_rs01_m01_ff_arch option (to archive fieldsfiles) being set to true, which I presume means it is just not *producing* anything, so there's nothing to archive. I'm not really sure what's going on - I have quickly checked against another nesting suite I have at vn 10.4 (u-aw620) and I can't see any differences in the GUI. Is there an obvious switch that I have forgotten to turn on?

Cheers,
Ella

comment:6 Changed 2 months ago by willie

Hi Ella,

Just a wild guess, but could you try setting verstash to false on the config 1 set up page. If it works then, it must be something to do with verstash.

Willie

comment:7 Changed 2 months ago by shakka

Hi Willie,

That hasn't worked either, so I'm a bit stumped. Sorry I can't be more helpful!

Ella

comment:8 Changed 7 weeks ago by willie

Hi Ella,

I have reverted your changes to

M       app/um/opt/rose-app-verstash.conf
M       app/um/rose-app.conf

and switched off the 1.5km domain. Still the 4km model produces empty LBC STASH.

I also took a copy of the 11.1 standard Nesting Suite (u-av356) and added your branch to it. This does produces good LBC STASH in the 4km domain.

There are not many changes left against the earlier u-ba903@89256 (assuming that worked)

  • In suite-adds.rc the constant in calculating the hyper threads has been changed from 3 to 6.
  • numerous changes in rose-suite.conf, but these are mainly due to the change in domain.

I notice that when the suite is running, the Peninsula_1p5km_nest_test_um_frame tasks all fail which means that the first failure does not stop running the remaining tasks. This suggests something might be wrong with the task graph in suite.rc file.

It might be an idea to start afresh and just set up the global and 4km LAM to make sure that it does produce the correct LBC STASH.

Regards
Willie

comment:9 Changed 7 weeks ago by shakka

Hi Willie,

Thanks for looking into this for me. I changed the hyper threading following another ticket when my glm recon kept failing, which solved that problem.

I'll start again with the 4 km LAM and see if that solves it, thanks.

Ella

comment:10 Changed 4 weeks ago by willie

Hi Ella,

Is this still a problem?

Willie

comment:11 Changed 4 weeks ago by shakka

Hi Willie,

I restarted again from scratch and that run has worked ok. Not sure what was wrong with the original suite, but you can close this ticket now, thanks.

Ella

comment:12 Changed 4 weeks ago by willie

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.