#2673 closed help (fixed)

Temporary increase in RDF quota

Reported by: charlie Owned by: grenville
Component: Disk Space Keywords:
Cc: Platform: ARCHER
UM Version:

Description

Hi,

Would it be possible to have yet another increase (temporary this time) to my quota on the RDF? I realise I already have a huge amount of data on there, and my quota is currently 70T which is higher than most people. The reason I ask, however, is because I am currently transferring most of my data to JASMIN, to be archived. Transferring the files individually using rsync is not working, because there are so many of them it times out. Even transferring a decade at a time (of my 400 year simulation) is too much.

Andy H recommended I tar up the files first to make the transfer easier, but I can't do this for anything meaningful (e.g. a decade at a time) because I don't have enough space left in my quota (I'm currently at ~69.5T). Each year when tarred is ~93G, so to do all 400 years (either tarring individually, or per decade or whatever) I would need at least another ~37T.

I realise this is a big ask, but I can't think of any other way to transfer my data.

Unless, of course, you know of a better way to transfer large amounts of data?

Many thanks

Charlie

Change History (10)

comment:1 Changed 11 months ago by grenville

Increased to 110TB

Grenville

comment:2 Changed 11 months ago by grenville

Charlie

Have a look at this

Globus Online https://www.globus.org/app/transfer) provides a managed service to orchestrate and monitor transfers between gridftp endpoints in a more user-friendly way, so is recommended as an alternative to setting up third-party transfers manually. See data-transfer-tools-gridftp-using-globus-online (http://help.ceda.ac.uk/article/3807-data-transfer-tools-gridftp-using-globus-online)

Grenville

comment:3 Changed 11 months ago by charlie

Thanks very much Grenville.

I have now had a look at Globus Online and, following the various instructions, am now practising transferring a file from the RDF to the appropriate place on JASMIN. It appears to be working, so far!

Assuming this works, is it still better to tar up each of my folders (of which there are 2 folders per year, for 400 years) before transferring, which is what Andy H advised I do? Or does it not matter using this way?

Charlie

comment:4 Changed 11 months ago by charlie

Also, further to this, it has just finished transferring one of my directories (i.e. 6 months worth of model data, approximately 30G's worth). It took just over 5 minutes. Is this about right?

comment:5 Changed 11 months ago by grenville

Hi Charlie

I don't think there's any need to tar your files - Globus should be smart enough to manage the data. 30GB in 5 min is 100MB/s — that is pretty quick.

Grenville

comment:6 Changed 11 months ago by charlie

Very many thanks. I started it off transferring another hundred years this morning, and it appears to be working (roughly 25% done). The green progress bar occasionally goes orange, saying it has timed out, but it appears to resolve itself. Indeed, at the moment it is orange, saying "credentials expiring soon", but it still appears to be transferring. Do I need to worry about these warnings?

comment:7 Changed 11 months ago by charlie

I have just actually answered that last question: yes. It told me that my credentials on the JASMIN side have now expired, giving instructions on how to reactivate them. I have just done this, and my transfer has restarted. Very clever! Do you know what it means by this, and how often it is likely to happen?

comment:8 Changed 11 months ago by grenville

I can't remember how long the credentials last - that's a question for CEDA. Glad it's working - it's designed to manage interruptions so you don't have to.

comment:9 Changed 11 months ago by charlie

Many thanks.

comment:10 Changed 11 months ago by charlie

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.