Opened 12 years ago

Closed 12 years ago

#166 closed help (fixed)

problems with ff2pp on hector for re-initialized um output

Reported by: sws07sib Owned by: jeff
Component: UM Tools Keywords:
Cc: Platform:
UM Version:

Description

Hi,

I try to convert my 64 bit Um output to 32 bit pp output on hector in order to work with the files on my local machine. The Um output is not normal pp output, but since I output loads of data for a sensitivity study, I had to reinitialize the pp-files (option inb the UM). Now they are called like xdjkfa_pj066. Does ff2pp not work with those files?

I also tried ieee -32 umfile umfile.32bit
convpp umfile.32bit umfile.pp, but then convpp crashes with

GCOM Version 3.0
Non MPP Single Processor
Using precision : 64bit INTEGERs and 64bit REALs
Built at Tue Oct 23 18:04:45 BST 2007
=====================================================


InitPrintStatus?: PrintStatus? initialised= 3

FILE STATUS
===========

OPEN: File xdjkfa_pf066.32bit to be Opened on Unit 20 Exists

SETPOS: Seek Failed: Invalid argument
SETPOS: Unit 20 to Word Address -1 Failed with Error Code -1
Abort
Problem with CONVPP program

I am a bit clueless how to proceed at the moment. Do you have any hint for me?
The files I work with can be found here:

nid15875:sws07sib$ /work/n02/n02/sws07sib/xdjkf

Regards,

Sylvia

Change History (9)

comment:1 in reply to: ↑ description Changed 12 years ago by jeff

Hi Sylvia

Replying to sws07sib:

Hi,

I try to convert my 64 bit Um output to 32 bit pp output on hector in order to work with the files on my local machine. The Um output is not normal pp output, but since I output loads of data for a sensitivity study, I had to reinitialize the pp-files (option inb the UM). Now they are called like xdjkfa_pj066. Does ff2pp not work with those files?

Yes these are normal UM files and ff2pp should work.

I also tried ieee -32 umfile umfile.32bit
convpp umfile.32bit umfile.pp, but then convpp crashes with

GCOM Version 3.0
Non MPP Single Processor
Using precision : 64bit INTEGERs and 64bit REALs
Built at Tue Oct 23 18:04:45 BST 2007
=====================================================


InitPrintStatus?: PrintStatus? initialised= 3

FILE STATUS
===========

OPEN: File xdjkfa_pf066.32bit to be Opened on Unit 20 Exists

SETPOS: Seek Failed: Invalid argument
SETPOS: Unit 20 to Word Address -1 Failed with Error Code -1
Abort
Problem with CONVPP program

convpp won't work with 32 bit files, an extra executable needs to be installed on hector for this to work.

I am a bit clueless how to proceed at the moment. Do you have any hint for me?
The files I work with can be found here:

nid15875:sws07sib$ /work/n02/n02/sws07sib/xdjkf

I don't have permission to access this directory. If you run

chmod -R g+rX /work/n02/n02/sws07sib

that should fix the problem and I will take a look at your UM files.

Jeff.

comment:2 follow-up: Changed 12 years ago by sws07sib

Hi Jeff,

Thanks for looking into my data. I have changed permission according to your suggestion.

Cheers,

Sylvia

comment:3 in reply to: ↑ 2 Changed 12 years ago by sws07sib

Hi Jeff,

I figured out that my pp files seem to be o.k., when I copy the 64bit UM files from hector to my local machine here and then use ff2pp on my local machine here at the institute. But when I use ff2pp on hector and copy the resulting 32 bit pp-file to my local machine here, my idl scripts crash telling me the following, when I try to output the stashlist:

% READFDR: File xdjkga_pd060_32bit.pp does not have header records of the expected

length

% Error occurred at: READFDR 88 /opt/local/cgam/idl/old.ukmo_idl/lib/readfdr.p

ro

% SEARCH 156 /opt/local/cgam/idl/ukmo_wave/lib/search.pro
% PPALIST 30 /opt/local/cgam/idl/ukmo_wave/lib/ppalist.pro
% PPA 131 /opt/local/cgam/idl/ukmo_wave/lib/ppa.pro
% STASHLIST 49 /opt/local/cgam/idl/ukmo_wave/lib/stashlist.pr

o

% $MAIN$
% Execution halted at: $MAIN$

Did someone else have the same problem so far?

Cheers,

Sylvia

comment:4 follow-up: Changed 12 years ago by jeff

  • Status changed from new to assigned

Hi Sylvia

The problem is caused by the files on Hector being little endian and the ukmo idl library wanting big endian files. There are utilities available to byte swap the data, on Hector you can use

bigend -32 xdjkfa_pj066.pp xdjkfa_pj066_be.pp

and on the local system use

~jeff/bin/swapbytes_4 xdjkfa_pj066.pp xdjkfa_pj066_be.pp

You should run either of these commands after running ff2pp on Hector, then the files should work okay.

Jeff.

comment:5 in reply to: ↑ 4 Changed 12 years ago by sws07sib

Hi Jeff,

I converted the UM files using ff2pp and then applied

bigend -32 ${file} ${file:r}_be.pp on hector.

Still it seems that I can not work with the resulting files. When opening the resulting bigendian 32bit pp files on my local system, the x and y-coordinates are constant now. Is there something I am missing? I have converted files loads of times on the hpcx and opened them on my local machine and it always worked so far.

Sylvia

Replying to jeff:

Hi Sylvia

The problem is caused by the files on Hector being little endian and the ukmo idl library wanting big endian files. There are utilities available to byte swap the data, on Hector you can use

bigend -32 xdjkfa_pj066.pp xdjkfa_pj066_be.pp

and on the local system use

~jeff/bin/swapbytes_4 xdjkfa_pj066.pp xdjkfa_pj066_be.pp

You should run either of these commands after running ff2pp on Hector, then the files should work okay.

Jeff.

comment:6 Changed 12 years ago by sws07sib

Hi Jeff,

it seems as if I can convert the data on my local machine, but not on hector by copying the UM 64 bit files onto my local system. Then swap the bytes order using your swapbytes_8 and then using ff2pp. Still it would be interesting for me to know, why it does not work on hector, since copying the 64 bit UM files onto the local system takes some time. But for the moment I think I have found a solution and can at least plot my data now.

Cheers,

Sylvia

comment:7 Changed 12 years ago by jeff

Hi Sylvia

I think I understand what is happening now. The reason it always works on hpcx is that is a big endian machine so no byte swapping is needed.

When I run ff2pp on your xdjkfa_pj066 file I get this output

file xdjkfa_pj066 is a Cray um file

This is wrong, ff2pp tries to guess the file type and in this case gets it wrong, the correct file type should be "byte swapped 64 bit ieee um file". This is the reason your pp file is incorrect whether you byte swap or not. Interestingly if you byte swap xdjkfa_pj066 first then ff2pp will guess the file type correctly, that is why it works when you do that locally. There is an option (-ieee64bs) to ff2pp to tell it the input file type, if you use that everything seems to work. Here is what you need to do on hector

ff2pp -ieee64bs xdjkfa_pj066 xdjkfa_pj066_le.pp
bigend -32 xdjkfa_pj066_le.pp xdjkfa_pj066.pp

I think this works, let me know either way.

Jeff.

comment:8 Changed 12 years ago by sws07sib

Hi Jeff,

Many thanks for your help. I just tried it out on hector and it seems to work.

Thanks again,

Sylvia

comment:9 Changed 12 years ago by jeff

  • Resolution set to fixed
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.