Opened 6 months ago

Last modified 5 months ago

#2011 accepted help

Automatic archiving to /nerc

Reported by: markmuetz Owned by: ros
Priority: normal Component: Archiving
Keywords: Cc:
Platform: ARCHER UM Version: 10.5

Description

How can I turn on/enable automatic archiving to e.g. /nerc/n02/n02/mmuetz/um10.5_runs/<run_name> after a UM run has finished? I would like it to do a complete copy of the work directory. As a bonus, how would I go about adding in post processing, e.g. pp → nc, once it's been archived (or before it's been archived if this is easier)?

Cheers,

Mark

Attachments (1)

suite.rc (7.8 KB) - added by markmuetz 5 months ago.

Download all attachments as: .zip

Change History (5)

comment:1 Changed 5 months ago by ros

Hi Mark,

Sorry for the delay in responding to this. If you simply wish to move/copy specific files from the <suite-id>/work directories to /nerc then the built in Rose app rose-arch might be all that you need.

I'm currently in the process of modifying the standard Met Office post-processing app to archive files on ARCHER from /work to /nerc including an option to convert fieldsfiles to pp before they are archived.

If you wanted to then convert from pp → nc then you'd probably need to either modify my post-processing app or write a new app just to do the conversion to netCDF after the archive has run.

Cheers,
Ros

comment:2 Changed 5 months ago by ros

  • Owner changed from um_support to ros
  • Status changed from new to accepted

Changed 5 months ago by markmuetz

comment:3 Changed 5 months ago by markmuetz

Hi Ros,

Sorry for the delay in getting back. I'm not sure what you mean by using the Rose app, can you point me towards info on how to do this?

What I've tried is to edit suite.rc, adding in two new jobs: [[archive]] and [[post-proc]], and added relevant parts to the cylc dependency graph. It works, but it's a little bit ugly and relies on hard coded paths. I've attached my suite.rc file (see the sections mentioned above, I'm not sure how to use the env vars to reference e.g. the ARCHER work dir, or the nerc dir), perhaps we could sit down at some point this week and try to work out how to make it a bit better (or just use the rose-arch app if this is better)?

Cheers,
Mark

comment:4 Changed 5 months ago by ros

Hi Mark,

Sorry again for delay in getting back to you. You can find information on the built-in rose-arch app here: http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-arch.html

Happy to talk to you more about rose-arch if you feel it is something that would work for you.

I'll take a look at your suite.rc today or tomorrow and then perhaps we could chat next week about the best way forward - Tuesday or Wednesday any good?

Cheers,
Ros.

Note: See TracTickets for help on using tickets.