Unable to submit suite because directory isn't empty, when it is!

[charlie]

Hi all,

Sorry to bother you again, but please can someone tell me what JULES/JASMIN is playing at this time?!

I'm trying to resubmit my suite, just like I have successfully done several times before. I have removed my cylc-run directory on PUMA, and have made sure that there are no jules.exe processes running still. I have logged out and logged in again, both PUMA and JASMIN. When I submit my suite, however, I get:

[FAIL] [FAIL] 2017-06-30T17:47:45+0100 [Errno 39] Directory not empty: 'log.20170630T120510Z/job/1/jules/01'

If I log onto JASMIN and try to remove the entire cylc-run directory as well, I can't:

[cwilliams2011@jasmin-sci2 ~]$ rm -r cylc-run/
rm: cannot remove `cylc-run/u-am232/log.20170630T120510Z/job/1/jules/01': Directory not empty

However, if I look in this directory, it IS empty:

[cwilliams2011@jasmin-sci2 ~]$ ls cylc-run/u-am232/log.20170630T120510Z/job/1/jules/01
[cwilliams2011@jasmin-sci2 ~]$

and if I try to remove whatever it thinks is in this directory, it tells me it is empty:

[cwilliams2011@jasmin-sci2 ~]$ rm cylc-run/u-am232/log.20170630T120510Z/job/1/jules/01/*
rm: cannot remove `cylc-run/u-am232/log.20170630T120510Z/job/1/jules/01/*': No such file or directory

If I try to remove just this 01 directory, it again tells me it's not empty:

[cwilliams2011@jasmin-sci2 ~]$ rm -r cylc-run/u-am232/log.20170630T120510Z/job/1/jules/01
rm: cannot remove `cylc-run/u-am232/log.20170630T120510Z/job/1/jules/01': Directory not empty

What's going on this time, and why does JASMIN hate me so much?!

Charlie

[charlie]

Hi all,

Further to the above, it appears to have fixed itself over the weekend. It now submits fine. Any ideas as to what happened?

Charlie

[ros]

Hi Charlie,

It's impossible to say, but the symptoms you describe are similar to those I've occasionally encountered on Monsoon where the filesystem is exceedingly slow to release a file lock.

Glad to hear it's sorted itself out over the weekend.

Cheers,
Ros.