submit fails

[awright]

Hello Patrick,

My jules run (u-bh383-NOTRIFFID) keeps failing due to the 'submit failed' error. The number of tasks that succeed in each attempt is different. For the same model run I can have 53 succeeded tasks in one attempt and 1 succeeded tasks in another attempt.

All the best,

Azin

[pmcguire]

Hi Azin
Did you look at your log files? I looked at several of your log files, and found a couple in the most recent run that had error messages.
The error message I found (~azin/cylc-run/log/job/1/jules_cg_tch_presc0/03/job.err) was, for example:
/home/users/azin/.lsbatch/1557226085.5080818: line 8: cannot create temp file for here-document: No space left on device.

I am not sure which hard drive it was that was running short on space. It could be your home directory. It could be the /work/scratch directory. It could be the /tmp directory. One way to check space is with pan_df -H directory_name.

If it's the /tmp directory, then it might be helpful to add export TMPDIR='/home/users/azin/tmpdir' to your .bashrc file and then do a mkdir /home/users/azin/tmpdir from the command line, in order to redefine the /tmp directory to a place where you have more control over. We're not supposed to be using the /tmp directory for most things anyways.

Or maybe you have run afoul of your 100GB quota on the /home directory? I was told that there were 100GB quotas on JASMIN for the home directory (the quotas used to be 10 GB), but I have never reached the 100GB quota myself. You can do a du -sck * from your home directory to see how many kilobytes you have used there. I am not sure if the quota command works completely.

Does this help?
Patrick

[awright]

Hello Patrick,

I cleared up some space and it is working fine now. Thank you very much!

All the best,

Azin

[pmcguire]

Excellent, Azin!
I will close the ticket now.
Patrick