Opened 12 months ago

Closed 4 months ago

#2011 closed help (answered)

Automatic archiving to /nerc

Reported by: markmuetz Owned by: ros
Priority: normal Component: Archiving
Keywords: Cc:
Platform: ARCHER UM Version: 10.5

Description

How can I turn on/enable automatic archiving to e.g. /nerc/n02/n02/mmuetz/um10.5_runs/<run_name> after a UM run has finished? I would like it to do a complete copy of the work directory. As a bonus, how would I go about adding in post processing, e.g. pp → nc, once it's been archived (or before it's been archived if this is easier)?

Cheers,

Mark

Attachments (1)

suite.rc (7.8 KB) - added by markmuetz 11 months ago.

Download all attachments as: .zip

Change History (6)

comment:1 Changed 11 months ago by ros

Hi Mark,

Sorry for the delay in responding to this. If you simply wish to move/copy specific files from the <suite-id>/work directories to /nerc then the built in Rose app rose-arch might be all that you need.

I'm currently in the process of modifying the standard Met Office post-processing app to archive files on ARCHER from /work to /nerc including an option to convert fieldsfiles to pp before they are archived.

If you wanted to then convert from pp → nc then you'd probably need to either modify my post-processing app or write a new app just to do the conversion to netCDF after the archive has run.

Cheers,
Ros

comment:2 Changed 11 months ago by ros

  • Owner changed from um_support to ros
  • Status changed from new to accepted

Changed 11 months ago by markmuetz

comment:3 Changed 11 months ago by markmuetz

Hi Ros,

Sorry for the delay in getting back. I'm not sure what you mean by using the Rose app, can you point me towards info on how to do this?

What I've tried is to edit suite.rc, adding in two new jobs: [[archive]] and [[post-proc]], and added relevant parts to the cylc dependency graph. It works, but it's a little bit ugly and relies on hard coded paths. I've attached my suite.rc file (see the sections mentioned above, I'm not sure how to use the env vars to reference e.g. the ARCHER work dir, or the nerc dir), perhaps we could sit down at some point this week and try to work out how to make it a bit better (or just use the rose-arch app if this is better)?

Cheers,
Mark

comment:4 Changed 10 months ago by ros

Hi Mark,

Sorry again for delay in getting back to you. You can find information on the built-in rose-arch app here: http://metomi.github.io/rose/doc/rose-rug-advanced-tutorials-arch.html

Happy to talk to you more about rose-arch if you feel it is something that would work for you.

I'll take a look at your suite.rc today or tomorrow and then perhaps we could chat next week about the best way forward - Tuesday or Wednesday any good?

Cheers,
Ros.

comment:5 Changed 4 months ago by grenville

  • Resolution set to answered
  • Status changed from accepted to closed

closed for lack of activity

Note: See TracTickets for help on using tickets.