Opened 8 weeks ago

Closed 8 weeks ago

#3262 closed help (answered)

Suite won't restart

Reported by: charlie Owned by: um_support
Component: UM Model Keywords:
Cc: Platform: NEXCS
UM Version: 10.7

Description

Hi,

Sorry to bother you, but one of my suites (that has been perfectly stable till now) appears to have failed over the weekend. I have tried shutting down and restarting it, but get the error below. Can you advise on what has happened here? Might it be something to do with a storage issue (i.e. I have too much), either on NEXCS and/or JASMIN, as I haven't done any tidying/archiving for several days? Or is there a problem with the machine itself?

Thanks,

Charlie

cwilliams@xcslc1:~/roses/u-br871> rose suite-run --restart
[INFO] export CYLC_VERSION=7.8.3
[INFO] export ROSE_ORIG_HOST=xcslc1
[INFO] export ROSE_SITE=
[INFO] export ROSE_VERSION=2019.01.2
[INFO] delete: log/rose-suite-run.conf
[INFO] symlink: rose-conf/20200512T105028-restart.conf <= log/rose-suite-run.conf
[INFO] delete: log/rose-suite-run.version
[INFO] symlink: rose-conf/20200512T105028-restart.version <= log/rose-suite-run.version
[INFO] chdir: log/
[FAIL] cylc restart u-br871 # return-code=1, stderr=
[FAIL] Traceback (most recent call last):
[FAIL]   File "/common/fcm/cylc-7.8.3/bin/cylc-restart", line 25, in <module>
[FAIL]     main(is_restart=True)
[FAIL]   File "/common/fcm/cylc-7.8.3/lib/cylc/scheduler_cli.py", line 134, in main
[FAIL]     scheduler.start()
[FAIL]   File "/common/fcm/cylc-7.8.3/lib/cylc/scheduler.py", line 237, in start
[FAIL]     self.suite_db_mgr.restart_upgrade()
[FAIL]   File "/common/fcm/cylc-7.8.3/lib/cylc/suite_db_mgr.py", line 524, in restart_upgrade
[FAIL]     pri_dao.vacuum()
[FAIL]   File "/common/fcm/cylc-7.8.3/lib/cylc/rundb.py", line 1031, in vacuum
[FAIL]     return self.connect().execute("VACUUM")
[FAIL] sqlite3.OperationalError: database is locked

Change History (4)

comment:1 Changed 8 weeks ago by grenville

Charlie

Pleae clean up your files; nexcs-n02 was at 100% capacity last night. Several job have failed as a result.

Grenville

comment:2 Changed 8 weeks ago by charlie

Apologies Grenville. I will clear up some space. From looking at it quickly, most of my stuff on /projects/nexcs-n02/cwilliams is actually within cylc-run, not (as I suspected) within sweet (which although being my output directory, only actually contains 10T). Am I right in thinking that all the suites in cylc-run here are simply copies of those in cylc-run of my home directory, and are therefore not actually needed (especially if I no longer need that suite)?

comment:3 Changed 8 weeks ago by grenville

Charlie

cylc-run under your home space just contains soft links to the cylc-run under /projects (check)

You need to clear up /projects to release space.

Grenville

comment:4 Changed 8 weeks ago by grenville

  • Resolution set to answered
  • Status changed from new to closed

nexcs-n02 disk looks much healthier - I'll close this ticket

Grenville

Note: See TracTickets for help on using tickets.