[Errno 16] Device or resource busy

[ggxmy]

Dear CMS,

I have been having a problem running a couple of my vn10.6.1 suites. I get error messages like below;

myosh@exvmsrose:u-al556 $ rose suite-run
[INFO] create: log.20170504T084117Z
[INFO] delete: log
[INFO] symlink: log.20170504T084117Z <= log
[INFO] log.20170503T161917Z.tar.gz <= log.20170503T161917Z
[FAIL] [Errno 16] Device or resource busy: 'log.20170503T161917Z/.nfs0380009155019bdb00002865'

I found this page
http://cms.ncas.ac.uk/wiki/RoseCylc/Hints#Deviceorresourcebusywhenrunningsuite

saying "You have one of the output files open somewhere, which means rose can't archive the old output. Close the file." but I can not find any file open.

The file log.20170503T161917Z/.nfs0380009155019bdb00002865 cannot be deleted even with \rm -f;

myosh@exvmsrose:u-al556 $ pwd
/home/d03/myosh/cylc-run/u-al556
myosh@exvmsrose:u-al556 $ ll log.20170503T161917Z/.nfs0380009155019bdb00002865
-rw-r--r--. 1 myosh mo_users 0 May  3 16:19 log.20170503T161917Z/.nfs0380009155019bdb00002865
myosh@exvmsrose:u-al556 $ \rm -f log.20170503T161917Z/.nfs0380009155019bdb00002865
rm: cannot remove `log.20170503T161917Z/.nfs0380009155019bdb00002865': Device or resource busy

The problem for u-al558 had gone after logging out and in to exvmsrose again. But I still have this problem for u-al556 after logging out and in many times. I have no other session to Monsoon open at this moment so any file must not be open.

Do you have any advice for this? Is it possible for you to delete the file above?

Thanks,
Masaru

[ros]

Hi Masaru,

The file should release when all the processes accessing it are closed but I can't see any processes of yours that might be accessing it. If I recall correctly you can move the directory in this state. If that doesn't work, as a temporary workaround till the file is released use the --no-log-archive option to rose suite-run so it doesn't try and tar up the log directory.

I'll pass this one onto the Monsoon team to look at.

Cheers,
Ros.

[ggxmy]

Hi Ros.,

OH, that's right. I forgot about that. I searched my emails and found that quick remedy. For future reference, that was to rename the entire job directory like this;

mv /home/d03/myosh/cylc-run/u-al556 /home/d03/myosh/cylc-run/u-al556.old

Thanks,
Masaru

[ros]

Hi Masaru,

The Monsoon team have deleted the NFS blocking file for you so you should now be able to delete the /home/d03/myosh/cylc-run/u-al556.old directory. The Rose & Cylc VMs haven't been rebooted since the migration to XCS so this will be done during the maintenance window on Tuesday to clear out any old NFS/file issues which should rectify this problem.

Regards,
Ros.

[ggxmy]

Hi Ros.,

Thank you for your help!

Masaru

[ros]

The Rose & Cylc VMs were rebooted yesterday. I shall close this query now, but if you see this problem again please let us and/or the Monsoon team know.

Cheers,
Ros.