Opened 2 years ago
Closed 2 years ago
#2673 closed help (fixed)
Temporary increase in RDF quota
Reported by: | charlie | Owned by: | grenville |
---|---|---|---|
Component: | Disk Space | Keywords: | |
Cc: | Platform: | ARCHER | |
UM Version: |
Description
Hi,
Would it be possible to have yet another increase (temporary this time) to my quota on the RDF? I realise I already have a huge amount of data on there, and my quota is currently 70T which is higher than most people. The reason I ask, however, is because I am currently transferring most of my data to JASMIN, to be archived. Transferring the files individually using rsync is not working, because there are so many of them it times out. Even transferring a decade at a time (of my 400 year simulation) is too much.
Andy H recommended I tar up the files first to make the transfer easier, but I can't do this for anything meaningful (e.g. a decade at a time) because I don't have enough space left in my quota (I'm currently at ~69.5T). Each year when tarred is ~93G, so to do all 400 years (either tarring individually, or per decade or whatever) I would need at least another ~37T.
I realise this is a big ask, but I can't think of any other way to transfer my data.
Unless, of course, you know of a better way to transfer large amounts of data?
Many thanks
Charlie
Change History (10)
comment:1 Changed 2 years ago by grenville
comment:2 Changed 2 years ago by grenville
Charlie
Have a look at this
Globus Online https://www.globus.org/app/transfer) provides a managed service to orchestrate and monitor transfers between gridftp endpoints in a more user-friendly way, so is recommended as an alternative to setting up third-party transfers manually. See data-transfer-tools-gridftp-using-globus-online (http://help.ceda.ac.uk/article/3807-data-transfer-tools-gridftp-using-globus-online)
Grenville
comment:3 Changed 2 years ago by charlie
Thanks very much Grenville.
I have now had a look at Globus Online and, following the various instructions, am now practising transferring a file from the RDF to the appropriate place on JASMIN. It appears to be working, so far!
Assuming this works, is it still better to tar up each of my folders (of which there are 2 folders per year, for 400 years) before transferring, which is what Andy H advised I do? Or does it not matter using this way?
Charlie
comment:4 Changed 2 years ago by charlie
Also, further to this, it has just finished transferring one of my directories (i.e. 6 months worth of model data, approximately 30G's worth). It took just over 5 minutes. Is this about right?
comment:5 Changed 2 years ago by grenville
Hi Charlie
I don't think there's any need to tar your files - Globus should be smart enough to manage the data. 30GB in 5 min is 100MB/s — that is pretty quick.
Grenville
comment:6 Changed 2 years ago by charlie
Very many thanks. I started it off transferring another hundred years this morning, and it appears to be working (roughly 25% done). The green progress bar occasionally goes orange, saying it has timed out, but it appears to resolve itself. Indeed, at the moment it is orange, saying "credentials expiring soon", but it still appears to be transferring. Do I need to worry about these warnings?
comment:7 Changed 2 years ago by charlie
I have just actually answered that last question: yes. It told me that my credentials on the JASMIN side have now expired, giving instructions on how to reactivate them. I have just done this, and my transfer has restarted. Very clever! Do you know what it means by this, and how often it is likely to happen?
comment:8 Changed 2 years ago by grenville
I can't remember how long the credentials last - that's a question for CEDA. Glad it's working - it's designed to manage interruptions so you don't have to.
comment:9 Changed 2 years ago by charlie
Many thanks.
comment:10 Changed 2 years ago by charlie
- Resolution set to fixed
- Status changed from new to closed
Increased to 110TB
Grenville