Opened 11 days ago

Last modified 3 days ago

#2465 new help

Puma: disk quotas

Reported by: mvguarino Owned by: um_support
Priority: normal Component: PUMA
Keywords: Cc:
Platform: PUMA UM Version:

Description

Hello,

When I tried to log into puma this morning I got this error message:

Disk quotas for user mvguarino (uid 1689): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
      /dev/sdb3 1042316* 1000000 1100000    none   24636       0       0        
sdb3: write failed, user block quota exceeded too long.

What should I do about it?

Thanks,

Vittoria

Change History (10)

comment:1 Changed 11 days ago by willie

Hi Vittoria,

You've run out of disc space on PUMA. Try deleting any files you no longer need. If you

du -mshc *

in your home directory you can see the directories taking the most space. You'll probably find that the contents of the cylc-run directory is the bulk user.

Regards
Willie

comment:2 Changed 11 days ago by mvguarino

Hi Willie,
In cleaning the cylc-run directory I accidentally removed the latest log directory, as a result my suite wouldn’t restart as it can’t find the rose-suite-run.conf file.
Is there a backup of the data on puma?

Thanks,

Vittoria

comment:3 Changed 11 days ago by grenville

Vittoria

What exactly is the error message?

Grenville

comment:4 Changed 11 days ago by mvguarino

Hi Grenville,

This is the error I get when I try to restart the suite:

[FAIL] [Errno 2] No such file or directory: '/home/mvguarino/cylc-run/u-as245/log/rose-suite-run.conf'

Indeed, while deleting the archived log files, I also deleted the cylc-run/u-as245/log directory on puma…
When I tried to recover it from ARCHER, I realized that in the ARCHER directory I only have the /job and /suites subdirs, and not all the other files that apparently the model needs to restart.

Vittoria

comment:5 Changed 11 days ago by grenville

Vittoria

We're checking the backups.

Grenville

comment:6 Changed 11 days ago by andy

Hi Vittoria,

The log files from the backup from last night are in /home/mvguarino/cylc-run/u-as245/log.20180309T104023Z.

Regards
Andy

comment:7 Changed 11 days ago by mvguarino

Thank you very much!

Vittoria

comment:8 Changed 3 days ago by mvguarino

Hello,
An update on this:
Although I deleted the old log files and some suites from my puma account, the disk quota error message is back, as a result the post processing of my suite (u-au022) is failing.

Could you please provide guidance on how to significantly and safely empty the cycl-run directory?

Thank you,

Vittoria

comment:9 Changed 3 days ago by ros

Hi Vittoria,

Andy has increased your quota on PUMA so you should be able to run ok now. Just keep an eye on the cylc-run directory so that it doesn't get too big.

You can safely delete any old log directories (or tar/gzip'd log directories from the cylc-run directories just make sure you keep the latest. Other than that if the log files keep filling up you can turn off the pulling of the log files back to PUMA for your suites.

Add to ~/.cylc/global.rc on PUMA:

[hosts]
   [[login\w*.archer.ac.uk]]
       retrieve job logs = False

or for an individual suite in the suite.rc (or archer.rc as appropriate), for example:

[[HPC]]
    ...
    [[[remote]]]
        host = $(rose host-select archer)
        ...
        retrieve job logs = False

Cheers,
Ros.

comment:10 Changed 3 days ago by mvguarino

Thank you,

I did modify the archer.rc file just to make sure it won't happen again any time soon.

Vittoria

Note: See TracTickets for help on using tickets.