Opened 11 years ago

Closed 11 years ago

#262 closed help (fixed)

output nan issue

Reported by: beh27 Owned by: willie
Component: UM Model Keywords: dump packing, topography, orography, Nans
Cc: Platform:
UM Version: 6.1

Description

I have been trying to get a LAM run with um6.1 running (xdwua: global with LBC output, xdwub: LAM). These are both based on the standard nested umui jobs. The LBC output from the global model looks fine so I'm happy that the first stage is running fine. The second job (xdwub) also runs fine, but there is a problem with the output. The .start file looks fine. I haven't changed the STASH so I'm getting 3 hourly output. For the wind data, for example, after the first three hours, NaNs? have taken over much of the domain leaving an area with data over a small area with high topography (center of Greenland) and data for the boundaries. After subsequent three hour dumps, only boundary data remains. This is the same for most other fields I've checked as well. I get the feeling that the data existing over high topography may indicate that the reconfiguration step with the topography is to blame, but if anyone has any experience of this kind of problem or can think of what is wrong then I'd be grateful to hear what they think!

Thanks, Benjamin

Change History (11)

comment:1 Changed 11 years ago by willie

  • Keywords topography, orography, Nans added
  • Owner changed from um_support to willie
  • Status changed from new to accepted
  • UM Version set to 6.1

Hi Benjamin,

It is possible that this due to mountains on the edge of the LAM. See ticket:34 which has a similar problem in a similar LAM. Chang said,

"The problem mainly caused by the gap between the scale of the orographies between the coarse and the inner domain. This gap induces an inconstancy in the lower level winds over locations where orography has a strong gradient in xy, xx or yy direction.

The possible solution are: 1) define a border away from strong orographic gradient; 2) decrease the time step of the model to solve rapid lower wind numerical instabilities."

Some users have succeeded by decreasing the timestep, and then repeatedly shifting the boundaries of the LAM to get as little steep orography on the boundaries of the domain/in the LBCs.

It is difficult for me to give a more accurate diagnosis: if you could give read permission to your files (esp the top one beh27) on home and work, then I could see some more detail.

Regards,

Willie

comment:2 Changed 11 years ago by beh27

Hi Willie,

Thanks for the advice, I will have a further play around with this in mind. I've given you read permission for my files now if you would like to root around to find any further issues…

Thanks,

Benjamin

comment:3 Changed 11 years ago by beh27

Another thought: I have just realised that the ticket you sent me to is that of my secondary supervisor. I actually used her job to run the model with the same LAM using one of her start files to check all my set up before running from my own start files. I actually ended up moving away from her setup and converting the UMUI standard nested LAM for the UK to the required domain over Greenland instead. This ran fine with her start files. The only difference now between this case and my new one is the change of start dump (and the fact that my start dump is in the new format, but we solved that problem) Is it just that the time at which the model becomes unstable is dependent on the case and specifically when there happen to be strong winds over high topographic gradient? I had a brief look at the gradient components in my ancil files and none of them seem to be particularly large near the boundaries. This seems to make sense considering my supervisor would have to have fiddled with the size of the domain to solve her problems with instabilities? Anyway, just a couple of thoughts…

Benjamin

comment:4 Changed 11 years ago by willie

Hi Benjamin,

Are you still having problems with this? I'm afraid I still can't see your files. Try

chmod -R a+rx /home/n02/n02/beh27
chmod -R a+rx /work/n02/n02/beh27

This will given every one read access to all your files.

Willie

comment:5 Changed 11 years ago by beh27

Hi Willie,

Yes, still having issues with this. Have now run another experiment (xdyl) with a different start dump but with the same outcome. I've tried playing with time steps and radiation timesteps, but without changing the output noticably

I've given you the permissions now, sorry, thought I had already done that…

thanks,

Benjamin

comment:6 Changed 11 years ago by willie

  • Keywords dump packing, added

Hi Benjamin,

This may be a packing issue. Got to Atmos > Control > Post Processing > Dumping and Meaning and select "unpacked primary and diagnostic" fields. I had previously thought that if the STASHmaster packing was switched off, it would be ok.

Regards

Willie

comment:7 Changed 11 years ago by beh27

Hi Willie,

Unfortunately this still doesn't work. I tried it out on job xdwub with no noticeable difference in the output. Any other thoughts?

Thanks,

Benjamin

comment:8 Changed 11 years ago by beh27

Hi Willie,

Any further ideas on a fix for this problem? I am quite keen to get this run going as soon as I can, I've been fiddling for quite a while, but I've come to a grinding halt in terms of what I know I can do to fix this. A couple of questions:

1) How close is the 7.1 nested LAM series to completion? I could feasibly put off this run until I can do it on 7.1, but only if its nearly done.

2) Stephen's almost identical problem (127) was solved by you reconfiguring the start dumps to the old met office format. I had a run working perfectly for these so could that be an option? I'd be able to give you a limited list of the start dumps i'd need altered if so.

Thanks for your continued support,

Benjamin

comment:9 Changed 11 years ago by beh27

Hi again,

Just a note to say that I've managed to get a run going (xdzkb, third start dump lucky!), but with the number of changes I made between jobs, I'm unsure of the reason why this third dump works. I will try out the previous two with exact copies of this new job. It could be as you previously suggested that the particular start conditions near high topography were producing instabilities or maybe the start dumps are just faulty?

Benjamin

comment:10 Changed 11 years ago by willie

Hi Benjamin,

I've had a look at this again - start dump 20081006_qwqg06.T+0, i.e. the original. I noticed that in the .leave file, the GCR was failing to converge. It would fail at time step three and then would not be required further, since the "RHS is zero" after that. I increased the timesteps per period from 576 to 14400, and now the GCR always converges, but I would suggest that its behaviour is not good. It converges within one iteration. After 140 time steps, the GCR is again no longer needed because the RHS is zero. Even so, the Nan's still appear in the U wind after the first time step. I am afraid I have run out of ideas. The fact that it is start dump dependant suggests that the whole scheme is marginally stable.

Regards,

Willie

comment:11 Changed 11 years ago by willie

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.