Opened 4 years ago

Closed 3 years ago

#1998 closed help (answered)

Transfer from ARCHER to JASMIN using rose suite

Reported by: cthomas Owned by: annette
Component: Rose Keywords: JASMIN, ARCHER, puma, rsync, scp, rose
Cc: Platform: PUMA
UM Version: <select version>



I am running a rose suite on ARCHER, launched from puma. At present I am manually transferring the output to JASMIN using rsync. I would be interested to know if this can be automated from inside the rose suite. Ideally, whenever a particular rose cycle finishes, rsync/scp would be launched and transfer the relevant directory (share/cycle/<date>).

I found this previous ticket:
Is that the sort of thing that could be put into a rose task?

It would also be good to:

  • zip the output before transferring it (hopefully easy - just run tar on the folder first)
  • relaunch the job if rsync fails for some reason (and provide a way to check this)


Change History (3)

comment:1 Changed 4 years ago by annette

  • Owner changed from um_support to annette
  • Status changed from new to assigned

Hi Chris,

Rose does have a built-in archiving system that you may want to look at:

However you may find that you want to just write your own script and add it as a task to be run every cycle.

Within cylc there is a mechanism for retrying a task a number of times if it fails:

If you are just compressing the data to make the transfer smaller, and don't want the data to be stored this way, you can use the -z option to rsync.

If you want to submit the transfer task to run on JASMIN, we have a Rose/cylc installation on the transfer nodes (jasmin-xfer[12]), under /home/users/aosprey/software. You just need to have the path set in your .profile or equivalent, i.e:

export PATH=/home/users/aosprey/software/bin:$PATH

What I would suggest is that you set up a simple rose suite with some dummy tasks and play around a bit.


comment:2 Changed 4 years ago by cthomas

Hi Annette,

Thanks, I will try some of those ideas. I would like to compress each relevant folder in order to avoid transferring thousands of small files to JASMIN. However, I don't want to interfere with the current cycle so need to think how to get it to run two cycles in the past. (Due to the setup of my suite, which could probably be improved, that's not entirely trivial.)

comment:3 Changed 3 years ago by grenville

  • Resolution set to answered
  • Status changed from assigned to closed
Note: See TracTickets for help on using tickets.