Opened 5 years ago

Closed 5 years ago

#1444 closed help (answered)

archiving to /nerc/n02/n02

Reported by: lsim Owned by: um_support
Component: Data Keywords:
Cc: Platform: ARCHER
UM Version: 4.5

Description

Dear CMS helpdesk,

I am running some HadCM3 v4.5.1 jobs on Archer and would like to automatically archive the output (from /work/n02/n02/lsim to the equivalent /nerc space).

I wrote a simple script to do this here: /home/n02/n02/lsim/bin/archiving.job

However when I try to run this as a cron job, by putting in crontab, it does not work. And the crontab data seems to have been removed when I checked, some hours later.

Is there another way of automatically archiving v4.5 output on Archer, or is there something that I am missing in trying to set up my cron job?

Best wishes,
Louise

Change History (5)

comment:1 Changed 5 years ago by grenville

Louise

ARCHER doesn't allow cron jobs. We have not implemented automatic archiving in UM vn4.5, and don't propose to given the number of UM versions we support.

I can only suggest that data be transferred to /nerc after the run has completed. Several projects have adopted this approach (transferring 100's TB of data) without difficulty.

What data volumes will you need to transfer?

Please change the permissions on your home and work spaces so that we have read access.

Grenville

comment:2 Changed 5 years ago by lsim

Hi Grenville,

Thanks for information re: cron jobs. And sorry about permissions! Changing that now…

I currently seem to have 100Gb of workspace. Due to model instability I output a lot of startdumps (to restart from). This fills up the workspace quite rapidly (over ~24 hours) when a couple of jobs are running. So in particular, my continuation runs then need daily interaction i.e. me to come along and type ~/bin/archiving.job, which is unfortunate, particularly at the weekend.

I will reduce the number of restart dumps a bit to help alleviate the issue. But would it also be possible to increase my workspace allocation?

Cheers,
Louise

comment:3 Changed 5 years ago by grenville

Louise

I bumped up your /work allocation to 2TB (it may take a few hrs to become available). We'll get back to you about the ancillary file problem.

Grenville

comment:4 Changed 5 years ago by lsim

Great! Thank you.
Louise

comment:5 Changed 5 years ago by grenville

  • Resolution set to answered
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.