Opened 7 months ago

Closed 7 months ago

#2898 closed help (fixed)

submit fails

Reported by: awright Owned by: pmcguire
Component: JULES Keywords: jasmin, jules
Cc: Platform: JASMIN
UM Version:

Description

Hello Patrick,

My jules run (u-bh383-NOTRIFFID) keeps failing due to the 'submit failed' error. The number of tasks that succeed in each attempt is different. For the same model run I can have 53 succeeded tasks in one attempt and 1 succeeded tasks in another attempt.

All the best,

Azin

Attachments (1)

Screen Shot 2019-05-07 at 12.29.28.png (92.0 KB) - added by awright 7 months ago.

Download all attachments as: .zip

Change History (5)

Changed 7 months ago by awright

comment:1 Changed 7 months ago by pmcguire

  • Status changed from new to accepted

Hi Azin
Did you look at your log files? I looked at several of your log files, and found a couple in the most recent run that had error messages.
The error message I found (~azin/cylc-run/log/job/1/jules_cg_tch_presc0/03/job.err) was, for example:
/home/users/azin/.lsbatch/1557226085.5080818: line 8: cannot create temp file for here-document: No space left on device.

I am not sure which hard drive it was that was running short on space. It could be your home directory. It could be the /work/scratch directory. It could be the /tmp directory. One way to check space is with pan_df -H directory_name.

If it's the /tmp directory, then it might be helpful to add export TMPDIR='/home/users/azin/tmpdir' to your .bashrc file and then do a mkdir /home/users/azin/tmpdir from the command line, in order to redefine the /tmp directory to a place where you have more control over. We're not supposed to be using the /tmp directory for most things anyways.

Or maybe you have run afoul of your 100GB quota on the /home directory? I was told that there were 100GB quotas on JASMIN for the home directory (the quotas used to be 10 GB), but I have never reached the 100GB quota myself. You can do a du -sck * from your home directory to see how many kilobytes you have used there. I am not sure if the quota command works completely.

Does this help?
Patrick

comment:2 Changed 7 months ago by awright

Hello Patrick,

I cleared up some space and it is working fine now. Thank you very much!

All the best,

Azin

comment:3 Changed 7 months ago by pmcguire

Excellent, Azin!
I will close the ticket now.
Patrick

comment:4 Changed 7 months ago by pmcguire

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.