Opened 21 months ago
Closed 21 months ago
#2898 closed help (fixed)
submit fails
Reported by: | awright | Owned by: | pmcguire |
---|---|---|---|
Component: | JULES | Keywords: | jasmin, jules |
Cc: | Platform: | JASMIN | |
UM Version: |
Description
Hello Patrick,
My jules run (u-bh383-NOTRIFFID) keeps failing due to the 'submit failed' error. The number of tasks that succeed in each attempt is different. For the same model run I can have 53 succeeded tasks in one attempt and 1 succeeded tasks in another attempt.
All the best,
Azin
Attachments (1)
Change History (5)
Changed 21 months ago by awright
comment:1 Changed 21 months ago by pmcguire
- Status changed from new to accepted
comment:2 Changed 21 months ago by awright
Hello Patrick,
I cleared up some space and it is working fine now. Thank you very much!
All the best,
Azin
comment:3 Changed 21 months ago by pmcguire
Excellent, Azin!
I will close the ticket now.
Patrick
comment:4 Changed 21 months ago by pmcguire
- Resolution set to fixed
- Status changed from accepted to closed
Note: See
TracTickets for help on using
tickets.
Hi Azin
Did you look at your log files? I looked at several of your log files, and found a couple in the most recent run that had error messages.
The error message I found (~azin/cylc-run/log/job/1/jules_cg_tch_presc0/03/job.err) was, for example:
/home/users/azin/.lsbatch/1557226085.5080818: line 8: cannot create temp file for here-document: No space left on device.
I am not sure which hard drive it was that was running short on space. It could be your home directory. It could be the /work/scratch directory. It could be the /tmp directory. One way to check space is with pan_df -H directory_name.
If it's the /tmp directory, then it might be helpful to add export TMPDIR='/home/users/azin/tmpdir' to your .bashrc file and then do a mkdir /home/users/azin/tmpdir from the command line, in order to redefine the /tmp directory to a place where you have more control over. We're not supposed to be using the /tmp directory for most things anyways.
Or maybe you have run afoul of your 100GB quota on the /home directory? I was told that there were 100GB quotas on JASMIN for the home directory (the quotas used to be 10 GB), but I have never reached the 100GB quota myself. You can do a du -sck * from your home directory to see how many kilobytes you have used there. I am not sure if the quota command works completely.
Does this help?
Patrick