#2604 closed help (fixed)

Problems in concatenating two NDdiag-processed PP files

Reported by: ambrogio Owned by: um_support
Component: UM Tools Keywords: PP files
Cc: Platform:
UM Version:

Description

Dear CMS Helpdesk,

I'm having problems in trying to concatenate two pp files that I have generated by using NDdiag.
The two files are called 20160610NDdiag_analysis.pp and 20160611NDdiag_analysis.pp and you can see them in Archer in this directory: /work/n02/n02/ambrogio/wmcginty_ticket_sep2018/ .
They contain some meteorological fields with hourly frequency for 24 hours on the day indicated (for this reason the last time step of the first file, 00z on 11/06/2016, is also the first time step of the second file).
As advised, I tried to concatenate them by typing cat 201606*.pp >> target.pp. The resulting file, that you can see in the same directory, has a size of 181.8GB (which is correct as the input files are 90.9GB each) but when opened with xconv or in python it shows only 25 time steps, from 00z 10/06 to 00z 11/06, which are the same of the first file (see screenshot attached).

Do you know what the problem could be?

Many thanks
Ambrogio

Attachments (1)

screenshot_concatenate.png (405.1 KB) - added by ambrogio 12 months ago.

Download all attachments as: .zip

Change History (5)

Changed 12 months ago by ambrogio

comment:1 Changed 12 months ago by willie

  • Keywords PP files added

Hi Ambrogio,

There is something wrong with one of the files:

ppinfo /work/n02/n02/ambrogio/wmcginty_ticket_sep2018/20160611NDdiag_analysis.pp | less

gives

file /work/n02/n02/ambrogio/wmcginty_ticket_sep2018/20160611NDdiag_analysis.pp is a byte swapped 32 bit ieee pp file 
Error in getting file type 
forrtl: error (76): IOT trap signal

The 20160610 file seems to be OK. How did you generate the second file?

Regards
Willie

comment:2 Changed 12 months ago by ambrogio

Dear Willie,

Thanks for pointing out this, as it helped me in finding out what the problem was.
As displayed in the message 20160611* is a bite swapped pp file whereas for some reason 20160610* is a simple pp file. Byte swapped pp files (i.e. big-endian) are the ones that are readable by Iris-python. Thus, I have swapped the endianness of 20160610* and now the concatenating process works fine ('cat' wasn't able to concatenate two files with different endianness).
Sorry for bothering with this. I should have realised by myself that there had been a glitch in the file endianness of one of the files!

Many thanks
Ambrogio

comment:3 Changed 11 months ago by ambrogio

Dear Willie,

A final update, as the previous one needed to be rectified.
Having made sure both files had the same endianness was of course necessary and allowed cat to run but still the result of the concatenation was not correct as cat got confused when seeing the same time instant in two different files. I also tried to use pp_merge to concatenate the two files without success.

Hence, the solution we decided to use was to delete the first time instant from each file (i.e. making the first file going from 01z 10/06 to 00z 11/06 and the second file from 01z 11/06 to 00z 12/06 to remove the overlap). To do that we used the um_subset tool (see here: http://home.badc.rl.ac.uk/iwi/um/utils.html#subset . Thanks Simon Wilson for pointing at it!). This tool works on um files (which I had used to generate the pp files I processed with NDdiag and then wanted to concatenate).

Now the concatenation process works fine so the ticket can be closed, thanks for the support!

Best wishes
Ambrogio

comment:4 Changed 11 months ago by willie

  • Resolution set to fixed
  • Status changed from new to closed

Ambrogio,

I am glad it works. Thanks for documenting the solution.

Regards
Willie

Note: See TracTickets for help on using tickets.