Opened 10 years ago

Closed 9 years ago

#343 closed help (fixed)

concatenating lbc files

Reported by: swr01hfd Owned by: willie
Component: UM Model Keywords: LBC files, concatenating, MergeData, VAR
Cc: willie Platform:
UM Version: 6.1

Description

Hi

I've generated three, 24 hour lbc files for successive days and I'd like to use them to perform a 3 day simulation. However, in the umui you can only specify 2 lbc files for any single job. Is there a way to concatenate lbc files? I've tried cat <file1> <file2> <file3> > <file4> but this doesn't work, I think because they're 64bit UM ancillary files and not pp files. I've also tried converting them all to pp files using ff2pp and then concatenating the files, but I get stuck when I try to convert the resulting file back into an ancillary file.

Any suggestions?

Cheers
Helen

Change History (15)

comment:1 Changed 10 years ago by jeff

  • Owner changed from um_support to jeff
  • Status changed from new to accepted

Hi Helen

I think the utility mergeum on hector should do what you want. On the NCAS-CMS web page under UM documents there is some information in documents F5 and F51. To combine your files I think you need to do something like

mergeum -t <file1> <file2> <newfile1>
mergeum -t <newfile1> <file3> <newfile2>

If this doesn't work you could always run the job in 2 or 3 chunks.

Jeff.

comment:2 Changed 10 years ago by swr01hfd

Hi Jeff

I've had a go with mergeum and am getting the error:

File 1 lookup record 314 matched with File 2 record 1
*ERROR* Length of model data specified wrongly : PARAMETER=419547752FILE=421617846FIXHD(161)419547752

Your initial dump may need reconfiguring.

CHK_LOOK: Consistency check - try reconfiguring your initial dump

It appears to be finding overlapping data in my two lbc files, which is good, but then it fails. I've had a look in the documentation you suggested but can't find out what's going wrong. Any ideas? The files I'm trying to merge are xeakd.alabcou1 and xeakq.alabcou1_ext, both of which are in directory /home/n02/n02/hdacre/work/xeakd.

Thanks
Helen

comment:3 Changed 10 years ago by jeff

  • Cc willie added

Hi Helen

I think this problem is caused by the 6.1 version of mergeum not being compiled with mods to allow it to work correctly on a little endian machine. I tried a later version which runs but doesn't support LBC fields. Looking at the comments in the source code it says

!LL  Purpose:
!LL          This program was primarily written to merge two boundary
!LL          datasets for use with the mesoscale model model in .
!LL          test mode.

But in the modification history it says

!     5.4  03/09/02    Arguments are added to DECOMPOSE_SMEXE.
!                      Check for landpt only and LBC fields
!                      - fields not supported         E.Leung

So it looks like this program no longer supports its primary purpose!

I've Cced willie into this query, maybe he has an idea how to do this. Running the model in 2 or 3 chunks maybe the best option.

Jeff.

comment:4 Changed 10 years ago by willie

Hi Helen,

I asked this question at the UM tutorial in the Met Office. There is a facility called MergeData? which is part of the VAR suite which can combine fields files. I can access this, so if you let me know where your files are, I can transfer them, combine them and send the results back.

regards,

Willie

comment:5 Changed 10 years ago by swr01hfd

Hi Willie

That's great. Do you know if it needs lbc files with overlapping times? If so, you can find 3 lbc files in directory /home/n02/n02/hdacre/work/xeakd. They are xeakd.alabcou1, xeakq.alabcou1_ext and xeakr.alabcou1_ext. Any chance we could get the facility on Hector, it would be useful for my student, Dan.

Cheers
Helen

comment:6 Changed 10 years ago by willie

  • Keywords LBC files, concatenating, MergeData, VAR added

Hi Helen,

I have now combined the files using the MergeData? tool in the Met Office's VAR suite. I had to run it on their HPC due to a 2Gbyte limit on the ocal Met Office machines. The result is on HECToR /work/n02/n02/wmcginty/willie2.alabcou1.

Some points of note. The original files overlapped in time slightly. So instead of a total of 77 fields there are 75 of one type (time?) and 2 of another. There is only one orography field. The best thing to do is try it and see. If it works, I will investigate getting the code onto HECToR.

Regards,

Willie

comment:7 Changed 10 years ago by swr01hfd

Hi Willie

I tried running with the concatenated lbc file and although the model runs for 3 days, the output fields aren't correct. They're fine for the first few outputs, i.e. until the second day. I'm not sure what's wrong as the lbc fields look fine but perhaps it would be a good idea to remove the extra fields in the lbc as a first try? Do you know how to do this, or should I generate lbc files with non-overlapping data?

Thanks
Helen

comment:8 Changed 10 years ago by willie

Hi Helen,

I can find no simple way of doing this. Could you generate the data without overlaps please.

Regards

Willie

comment:9 Changed 10 years ago by swr01hfd

Hi Willie

I've generated the lbc files with no overlaps. They can be found in directory /home/n02/n02/hdacre/work/xeakd and are:
231094_00Z-241094_02Z.alabcou1
241094_03Z-251094_02Z.alabcou1
251094_03Z-261094_02Z.alabcou1
261094_03Z-271094_02Z.alabcou1

Cheers
Helen

comment:10 Changed 10 years ago by jeff

  • Owner changed from jeff to willie
  • Status changed from accepted to assigned

comment:11 Changed 10 years ago by willie

Hi Helen,

The new file is ready in /work/n02/n02/wmcginty/Helen/willie3.alabcou1

regards,

Willie

comment:12 Changed 10 years ago by swr01hfd

Hi Willie

I've tried a run with the new merged lbc file. The output is fine for the first 24 hours, i.e. using the merged lbc's created from the first lbc file, but after 03Z on the second day, i.e. using the merged lbc's created from the second lbc file, the output is corrupted with lots of Nan's. To be precise the errors occur at timestep 122, which is 10hours and 10 mins after the start of the model run (16Z), i.e. at 02:10 on the second day.

I'm getting the following line in the .leave file lots.
RHS zero so GCR( 2 ) not needed
This usually means that a field somewhere has invalid values. My guess is that it's the lbc file. Any ideas?

Cheers
Helen

comment:13 Changed 10 years ago by willie

Hi Helen,

I think the LBC is OK. The problem occurs at 122 timesteps, which as you say is 10 hours 10 mins into the run; the transition between the original LBC's occurs much later at 27 hours into the run. Having said that, I note that the density field is huge, order 1013. Does this seem reasonable to you? And the W fields are always zero.

There seem to be at least three reasons for the lack of convergence:

  1. incorrect ancillary files,
  2. time step too large,
  3. large orographic gradient near the LAM boundary in the case of LAM runs.

I would try reducing the time step.

Regards,

Willie

comment:14 Changed 10 years ago by swr01hfd

Hi Willie

Actually, that's not strictly true. The first lbc is at 00Z, but the run doesn't start until 16Z, thus 122 timesteps corresponds to lbc at 26h 10mins, i.e. at the time when the transition between the original lbc's occurs.

Anyway, I'll give your suggestions a go.

Thanks
Helen

comment:15 Changed 9 years ago by willie

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.