#2371 closed help (fixed)

Start dump request

Reported by: a.elvidge Owned by: um_support
Component: Data Keywords:
Cc: Platform: Monsoon2
UM Version:

Description

Hi,

I would like to request some (a lot of) UM start dumps, please:
Every 24 hours at 00 UTC between and inclusive of 20151125T0000Z and 20160531T0000Z.

Would this be possible?

Thanks, Andy

Change History (12)

comment:1 Changed 20 months ago by willie

Hi Andy,
Yes it certainly possible. This is about 156 dumps and I think each are 24GB, so that is 3,744GB or about 3.7 TB, so you'll need to make sure you have space to store this.

I'll get started on this once I can access MASS - its down at the moment, due to recent problems affecting Monsoon and its systems.

Regards,
Willie

comment:2 Changed 20 months ago by a.elvidge

Hi Willie,
Do you know how I go about finding out how much space is available on my Monsoon account / project space?
Cheers, Andy

comment:3 Changed 20 months ago by willie

Hi Andy,

There is useful information at http://collab.metoffice.gov.uk/twiki/bin/view/Support/FileSystems.

Regards
Willie

comment:4 Changed 20 months ago by ros

Just to add, if I'm reading the information I have correctly, both polar and accacia currently have a 6Tb group quota. polar has used 24% of its quota and accacia 7%.

comment:5 Changed 20 months ago by willie

  • UM Version <select version> deleted

Hi Andy,

MASS is now back up and I'm starting to get the data. I miscalculated: it is actually 187 dumps and they are 22.84GB each, so 4.17TB in all. It takes 6.47 mins to get one dump, so 20.16 hours solid 'moo get' for them all. I am trying to get one months worth in one get to see if it's any faster.

Regards
Willie

comment:6 Changed 20 months ago by a.elvidge

Hi Willie,
That's quite a lot of data. Perhaps hold off for a bit. I'll have a think about alternatives, or whether to go ahead.
Cheers, Andy

comment:7 Changed 20 months ago by a.elvidge

Hi Willie, Ros,

Is my understanding correct that the start dumps are stored on MASS, and so are not part of the 6 gb allocated local disk space? In which case, please go ahead with the data transfer.

Thanks, Andy

comment:8 Changed 20 months ago by willie

Hi Andy,

Although they are stored in MASS they need to be extracted to be used. They will end up on, say, /projects/accacia/aelvidge, where they will use up quota.

For reference, extracting a months worth in one moo get is about four times faster than getting the files individually.

Let me know how you want to proceed.

Regards
Willie

comment:9 Changed 20 months ago by a.elvidge

Hi Willie,

Ok… the start dumps will only be there temporarily whilst I run the job. So I think it's ok to proceed, please.

Thanks, Andy

comment:10 Changed 20 months ago by willie

Hi Andy,

Your data is now ready on Monsoon in /projects/nexcs-n02/frmy/data_xfer. It is 189 dumps and 4.3TB.

Please copy it to your own work space as soon as possible and let me know when you've done that so I can free up my work space.

Regards
Willie

comment:11 Changed 19 months ago by a.elvidge

Hi Willie,

I have now copied all those start dumps across, thanks.

Cheers, Andy

comment:12 Changed 19 months ago by willie

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.