Opened 10 months ago

Closed 6 weeks ago

#1998 closed help (answered)

Transfer from ARCHER to JASMIN using rose suite

Reported by: cthomas Owned by: annette
Priority: normal Component: Rose
Keywords: JASMIN, ARCHER, puma, rsync, scp, rose Cc:
Platform: PUMA UM Version: <select version>

Description

Hi,

I am running a rose suite on ARCHER, launched from puma. At present I am manually transferring the output to JASMIN using rsync. I would be interested to know if this can be automated from inside the rose suite. Ideally, whenever a particular rose cycle finishes, rsync/scp would be launched and transfer the relevant directory (share/cycle/<date>).

I found this previous ticket: http://cms.ncas.ac.uk/ticket/1551
Is that the sort of thing that could be put into a rose task?

It would also be good to:

  • zip the output before transferring it (hopefully easy - just run tar on the folder first)
  • relaunch the job if rsync fails for some reason (and provide a way to check this)

Thanks,
Chris

Change History (3)

comment:1 Changed 10 months ago by annette

  • Owner changed from um_support to annette
  • Status changed from new to assigned

Hi Chris,

Rose does have a built-in archiving system that you may want to look at:
http://metomi.github.io/rose/doc/rose-rug-task-run.html#rose-task-run.util.rose_arch

However you may find that you want to just write your own script and add it as a task to be run every cycle.

Within cylc there is a mechanism for retrying a task a number of times if it fails: http://cylc.github.io/cylc/html/single/cug-html.html#12.14

If you are just compressing the data to make the transfer smaller, and don't want the data to be stored this way, you can use the -z option to rsync.

If you want to submit the transfer task to run on JASMIN, we have a Rose/cylc installation on the transfer nodes (jasmin-xfer[12]), under /home/users/aosprey/software. You just need to have the path set in your .profile or equivalent, i.e:

export PATH=/home/users/aosprey/software/bin:$PATH

What I would suggest is that you set up a simple rose suite with some dummy tasks and play around a bit.

Annette

comment:2 Changed 10 months ago by cthomas

Hi Annette,

Thanks, I will try some of those ideas. I would like to compress each relevant folder in order to avoid transferring thousands of small files to JASMIN. However, I don't want to interfere with the current cycle so need to think how to get it to run two cycles in the past. (Due to the setup of my suite, which could probably be improved, that's not entirely trivial.)
Chris

comment:3 Changed 6 weeks ago by grenville

  • Resolution set to answered
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.