Opened 3 months ago

Last modified 8 weeks ago

#3327 accepted help

postproc app

Reported by: amenon Owned by: ros
Component: UM Model Keywords: postproc app
Cc: Platform: ARCHER
UM Version: 10.9

Description

Hi,

I am trying to set up the data transfer from my suite in Archer to JASMIN as per http://cms.ncas.ac.uk/wiki/Archer/Transition2020/PPTransfer, but the suite doesn't have the postproc app. Could you please let me know how to add postproc to my suite. The suite id is u-bv556. Thanks.

Change History (8)

comment:1 Changed 3 months ago by ros

  • Owner changed from um_support to ros
  • Status changed from new to accepted

Hi Arathy,

This is a nesting suite which doesn't use UM archiving so very unlikely the pptransfer part from it will work either. Adding the entire postproc app is non-trivial. Can you point me to where you data is currently archived so I can take a look at its format.

It will probably be easier to just run a standalone transfer suite. I do have one that is in development that may work for you.

Regards,
Ros.

comment:2 Changed 3 months ago by amenon

Thanks Ros. I don't have any data archived in Archer now as I removed all of them. The following data in Jasmin is how the data will appear in the cylc directory:
/group_workspaces/jasmin2/incompass/shared/model/umnsaa_pa018
and the following is an example of an archived data from Archer RDF:
/group_workspaces/jasmin2/incompass/shared/model/WG4P4/20160630T0000Z_INCOMPASS_km4p4_RA1T_qcons_pb000.pp
Please let me know if you can't access them.

comment:3 Changed 3 months ago by ros

Hi Arathy,

I'm trying to run your suite. I've copied your ~/roses/u-bv556 directory on PUMA but it's failing on the reconfigurations as files are missing (e.g. $UMDIR/ancil/data/ancil_versions/n1280e/GA6.0/latest/ancils). Is there anything I need to know in order to get it to run? I also tried copying your start dump and the UM crashed.

Regards,
Ros.

comment:4 Changed 3 months ago by amenon

Hi Ros,

Thanks. You have to give a softlink to that ancil file in Archer:

ln -s /work/n02/n02/hum/ancil/data/ancil_versions/n1280e/ps39/latest/ancils /work/n02/n02/amenon/cylc-run/u-bv556/share/data/etc/ancil_versions_ga6

You might have to unlink it first. Hope this will solve that issue.

Regards,
Arathy

comment:5 Changed 8 weeks ago by amenon

Hi Ros,

I just wanted to check if you had any luck with getting the archiving work. I still couldn't get the LAM forecast of the first cycle to work yet as the suite keeps failing with NaNs? in error. I tried reducing the time step, but with no luck this time.

Thanks,
Arathy

comment:6 Changed 8 weeks ago by ros

Hi Arathy,

Sorry I got diverted. Looking at the suite and what you have on JASMIN, assuming you've not changed the directory structure from what was under /work/n02/n02/amenon/archive pptransfer won't work with that setup as it relies on data from each cycle being put in a separate directory.

Since I can't get the suite to run - km1p5 recon keeps timing out eventhough I've trebled the wallclock - I can't really do much more at present to see if rose_arch can be coerced into separating the archiving into separate cycles.

I suspect it will be easiest to run and then use a separate transfer suite to copy the data to JASMIN

If you need help sorting the NaNs out, I'd suggest opening a new ticket for that so we keep the 2 issues separate.

Cheers,
Ros.

comment:7 Changed 8 weeks ago by amenon

Thanks Ros. Then I will use a separate transfer suite to copy the data.Then one issue is that, with the current set up, the output file names are the same (eg., umnsaa_pa000, umnsaa_pa006 etc) for each cycle . When RDF was there, the output files were given the correct file name during the archiving step when it was being archived to the RDF. So before using the transfer suite, do I need to get the output file names right for each cycle? Another question is can I run the transfer suite after every few cycles as I can't store more data than that at a time in /work/n02/n02/amenon?

I will raise a separate ticket for NaNs? in error. I always had this issue with 1.5 km resolution. I will try to get one cycle out so that we can try the transfer suite.

comment:8 Changed 8 weeks ago by ros

Hi Arathy,

I'm a bit confused when you say the files were given the correct name when they were being archived to the RDF, the same archiving mechanism should be being used now just placing on a different path (/work rather than /nerc) so don't see why the file naming should be any different??

Cheers,
Ros.

Note: See TracTickets for help on using tickets.