Temporary increase in RDF quota

[charlie]

Hi,

Would it be possible to have yet another increase (temporary this time) to my quota on the RDF? I realise I already have a huge amount of data on there, and my quota is currently 70T which is higher than most people. The reason I ask, however, is because I am currently transferring most of my data to JASMIN, to be archived. Transferring the files individually using rsync is not working, because there are so many of them it times out. Even transferring a decade at a time (of my 400 year simulation) is too much.

Andy H recommended I tar up the files first to make the transfer easier, but I can't do this for anything meaningful (e.g. a decade at a time) because I don't have enough space left in my quota (I'm currently at ~69.5T). Each year when tarred is ~93G, so to do all 400 years (either tarring individually, or per decade or whatever) I would need at least another ~37T.

I realise this is a big ask, but I can't think of any other way to transfer my data.

Unless, of course, you know of a better way to transfer large amounts of data?

Many thanks

Charlie

[grenville]

Increased to 110TB

Grenville

[grenville]

Charlie

Have a look at this

Globus Online ​https://www.globus.org/app/transfer) provides a managed service to orchestrate and monitor transfers between gridftp endpoints in a more user-friendly way, so is recommended as an alternative to setting up third-party transfers manually. See data-transfer-tools-gridftp-using-globus-online (​http://help.ceda.ac.uk/article/3807-data-transfer-tools-gridftp-using-globus-online)

Grenville

[charlie]

Thanks very much Grenville.

I have now had a look at Globus Online and, following the various instructions, am now practising transferring a file from the RDF to the appropriate place on JASMIN. It appears to be working, so far!

Assuming this works, is it still better to tar up each of my folders (of which there are 2 folders per year, for 400 years) before transferring, which is what Andy H advised I do? Or does it not matter using this way?

Charlie

[charlie]

Also, further to this, it has just finished transferring one of my directories (i.e. 6 months worth of model data, approximately 30G's worth). It took just over 5 minutes. Is this about right?

[grenville]

Hi Charlie

I don't think there's any need to tar your files - Globus should be smart enough to manage the data. 30GB in 5 min is 100MB/s — that is pretty quick.

Grenville

[charlie]

Very many thanks. I started it off transferring another hundred years this morning, and it appears to be working (roughly 25% done). The green progress bar occasionally goes orange, saying it has timed out, but it appears to resolve itself. Indeed, at the moment it is orange, saying "credentials expiring soon", but it still appears to be transferring. Do I need to worry about these warnings?

[charlie]

I have just actually answered that last question: yes. It told me that my credentials on the JASMIN side have now expired, giving instructions on how to reactivate them. I have just done this, and my transfer has restarted. Very clever! Do you know what it means by this, and how often it is likely to happen?

[grenville]

I can't remember how long the credentials last - that's a question for CEDA. Glad it's working - it's designed to manage interruptions so you don't have to.

[charlie]

Many thanks.