Opened 6 years ago

Closed 5 years ago

#1301 closed help (fixed)

Submitting a standard CRUN

Reported by: charlie Owned by: ros
Component: UM Model Keywords:
Cc: Platform: ARCHER
UM Version: 6.6.3

Description

Hello,

Sorry to bother you again, but I seem to have the same problem as a couple of weeks ago (see ticket #1279, dealt with by Ros). I'm trying to resubmit one of my jobs as a standard CRUN, but get the following error:

FCM_MAIN: Submitting umuisubmit_run …
qsub: script file: No such file or directory
FCM_MAIN: Submit failed

Last time this happened, the reason was I had run out of space on /home. So it was upped. I have just checked on SAFE, however, and it's fine:

home (home2): Usage 10 Gb
home (home2): Quota 15 Gb

Please can you advise? I resubmitted my job as a CRUN a couple of days ago, and it worked fine - nothing has changed on Puma since.

Thanks a lot,

Charlie

Change History (8)

comment:1 Changed 6 years ago by ros

  • Owner changed from um_support to ros
  • Status changed from new to accepted

Hi Charlie,

Could you try submitting again today please? If you get the same problem I will have to forward this to ARCHER as it works fine for me and I can see nothing wrong with the disk space allocations.

Cheers,
Ros.

comment:2 Changed 6 years ago by charlie

Ros,

I have just resubmitted it again, and exactly the same problem I'm afraid. Sorry!

Charlie

comment:3 Changed 6 years ago by ros

Hi Charlie,

Can you please close down the UMUI completely and then run

puma$ export UMUI_SSH_DEBUG_LEVEL=5
puma$ umui > ~/umui.out

This ups the verbosity of all the ssh/scp commands and directs it all into ~/umui.out
Then resubmit your job. Let me know when you've done that and I'll examine the ~/umui.out file and see if that sheds some light on what's going on.

Cheers,
Ros.

comment:4 Changed 6 years ago by charlie

Okay, done, as requested.

Looks like a load of errors have appeared (gobbledygook to me, but no doubt understandable to you!)

Many thanks,

Charlie

comment:5 Changed 6 years ago by ros

Hi Charlie,

The problem is indeed a disk space issue. The output from the umui is riddled with "Disk quota exceeded" messages.

PRESM_A                                         0%    0     0.0KB/s   --:--ETA
PRESM_A                                       100% 4156     4.1KB/s   00:01
scp: umui_runs/xiogg-148113922/PRESM_A: Disk quota exceeded

A du -sh on your $HOME on ARCHER reveals you have used 16Gb which exceeds your quota.

ARCHER-xc30> du -sh cjrw09/
16G     cjrw09/

I have increased your disk quota again. I notice that 11Gb of your /home space is taken up by .leave files in ~/um/umui_out. Can you please remove those that you no longer need. I notice that ~200Mb of each .leave file is due to messages about "potential negative soil moisture". Are these relevant? If you continue to output these messages you will very quickly run out of disk space again.

I'm not sure why there is a mismatch between actual usage and what SAFE shows and will raise this with ARCHER.

Cheers,
Ros.

comment:6 Changed 6 years ago by charlie

Thanks very much, Ros - and I'm very sorry it was such a simple mistake. I didn't realise how big those .leave files were - I have now deleted them and resubmitted my job, and it's fine.

How can I turn off all the reporting of messages when I submit? I.e. undo UMUI_SSH_DEBUG_LEVEL=5

Charlie

comment:7 Changed 6 years ago by ros

Hi Charlie,

No problem - Archer SAFE didn't help the cause!!!

Easiest and best way to turn off the UMUI debug messages is to close down the UMUI, exit the PUMA terminal window and log back in again.

Otherwise closing down the UMUI, re-exporting UMUI_SSH_DEBUG_LEVEL=0 and then restarting the UMUI should also work.

Cheers,
Ros.

comment:8 Changed 5 years ago by ros

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.