Opened 3 years ago

Closed 3 years ago

#2001 closed help (fixed)

Moving files exceeds walltime limit

Reported by: Leighton_Regayre Owned by: um_support
Component: Other Keywords:
Cc: Platform:
UM Version:

Description

Hello,

I'm attempting to move a large number of files from /work to /nerc.

I move the files in batches using a script using the queue. It has worked perfectly in the past. Currently when attempting to move just 2 .pp files I get an error message:

walltime 27 exceeded limit 24

despite having entered a 24 hour limit. Could this be a /work space allocation issue?

The output file is here:
/work/n02/n02/lre/Dumps/mv2n_xmtna_sub_.e3994322

The /nerc space I'm attempting to move files to is here:
/nerc/n02/n02-CCPPE/lre

Change History (10)

comment:1 Changed 3 years ago by grenville

Leighton

Please change permissions on /work/n02/n02/lre/Dumps/mv2n_xmtna_sub_.e3994322 so I can read it.

Grenville

comment:2 Changed 3 years ago by Leighton_Regayre

Grenville,

I've changed permissions.

In case it's the cause of the problem the script I'm using to move the files is here:
/home/n02/n02/masara/bin/move2nerc.sh

Thanks,

Leighton.

comment:3 Changed 3 years ago by grenville

Leighton

Where is the output from move2nerc.sh, where is $subscript?

Grenville

comment:4 Changed 3 years ago by Leighton_Regayre

Grenville,

I use the default filepath:
/work/n02/n02/lre/Dumps

to move files to the directory:
/nerc/n02/n02-CCPPE/lre

The files I'm attempting to move at the moment have the root:
xmtn?
and contain:
*.pm*

There are only 2 of them with this root.

Leighton.

comment:5 Changed 3 years ago by grenville

Leighton

I'm not following this — I need to see the pbs script which was submitted to the machine. I'm not clear on why there is no /work/n02/n02/lre/Dumps/mv2n_xmtna_sub_.o3994322 file

I can't see which files you refer to here

The files I'm attempting to move at the moment have the root:
xmtn?
and contain:
*.pm*

Grenville

comment:6 Changed 3 years ago by Leighton_Regayre

Grenville,

Sorry for not explaining more fully.

The script requires me to enter the source and destination directories, followed by the folders (in my case xmtn*, meaning all directories starting with xmtn), then the type of file (*.pp).

The raw script is here:
home/n02/n02/masara/bin/move2nerc.sh

I'm not sure there's any /work/n02/n02/lre/Dumps/mv2n_xmtna_sub_.o3994322 file. That confuses me too.

Leighton.

comment:7 Changed 3 years ago by grenville

Leighton

Please run home/n02/n02/masara/bin/move2nerc.sh as you did previously and keep the file

subscript=$dirm2n/mv2n_${dirs[0]}_sub_$datim

That's the one I (think) I need to see.

Grenville

comment:8 Changed 3 years ago by grenville

Leighton

I see that some of your jobs have a wallclock time of 24 seconds —- could that be the problem?

Grenville

comment:9 Changed 3 years ago by Leighton_Regayre

Grenville,

Example output files are here:
/home/n02/n02/lre/umui_runs/scripts/move2nerc/mv2n_xmtoa_sub_20161019-083828

There was one submission where I used an alternative to the default wallclock because I was confused by an error message referring to a required 27 hours. This wasn't the cause of the transfer failing.

Today the jobs are copying fine using exactly the same submission process.

I'll let you know if the problem resurfaces.

Thanks for your help,

Leighton.

comment:10 Changed 3 years ago by ros

  • Component changed from UM Model to Other
  • Resolution set to fixed
  • Status changed from new to closed
  • UM Version <select version> deleted

No further comments received so closing this ticket.

Note: See TracTickets for help on using tickets.