Opened 4 years ago
Closed 4 years ago
#1988 closed help (answered)
archiving of log files in cylc-run directory
Reported by: | marcus | Owned by: | ros |
---|---|---|---|
Component: | UM Model | Keywords: | cylc-run, log files, archiving |
Cc: | Platform: | ARCHER | |
UM Version: | 10.4 |
Description
Hi, for each suite there are two versions of the cylc-run directory, one on Puma and one on ARCHER. I am trying to understand what happens to these files and where to find specific information.
- Puma seems to have a more complete set of log files, i.e. the directory structure in ~/cylc-run/{suite-name}/log/ contains more files, many of which are compressed into tar archives. ARCHER's version (linked to /work/n02/n02) has the model output under ~/cylc-rn/{suite-name}/share/ but not all of the log files. If I want to backup the information within the cylc-run directory at the end of the experiment to the RDF, should I backup the Puma version or the ARCHER version or both?
- Why does the job.status file get deleted when archiving with tar? The tar.gz files don't seem to contain this file any longer. This small file allows me to conveniently verify times of submission, initialisation and completion of the job, i.e. I can find out how long the job was held in the queue. Is there any other file which would let me track the submission or queue time of each cycle after the experiment has completed?
- What determines the timing when the ~/cylc-run/{suite-name}/log/ directory gets archived into a tar-file? Does this happen after a specific number of cycles and can this number be set somewhere in Rose?
Many thanks,
Marcus
Change History (7)
comment:1 Changed 4 years ago by ros
comment:2 Changed 4 years ago by marcus
Hi Ros,
This is really helpful to know. The only issue to understand is why the job.status file gets deleted in my runs. See, for instance, on puma
/home/marcus/cylc-run/u-ag352/log/job-19890901T0000Z.tar.gz
This file does no longer contain job.status. I wonder why, is this a setting in my configuration files?
Many thanks,
Marcus
comment:3 Changed 4 years ago by ros
- Owner changed from um_support to ros
- Status changed from new to accepted
Hi Marcus,
Ok. Sorry. We're talking about archiving of different log directories. I thought you meant the archiving of the whole ~/cylc-run/log directory not the ~/cylc-run/log/job sub-directories. These are controlled by different parts - the former is cylc the latter is a standard Rose app called rose_prune.
The tarring up of the job directory is controlled by the housekeeping app. The job directory is tarred up after the number of cycles specified by the variable archive-logs-at in the window housekeeping → prune There is no way to change what files it archives as far as I know. I'll talk to the Rose guys and see what it's supposed to do with the job.status file - the documentation doesn't note any exclusions.
Regards,
Ros.
comment:4 Changed 4 years ago by ros
Hi Marcus,
I've just been looking at the source code and it specifically excludes the job.status file. Why this is done I don't know yet. You could also extract the information, although a little more complicated from the job-activity.log
Cheers,
Ros
comment:5 Changed 4 years ago by marcus
OK, thank you Ros. I will have a look at the job-activity.log file (which I find very complicated to understand) in order to see where I can retrieve it. It seems odd that such a small file like job.status gets deliberately deleted. It can't be to save space, surely?
Many thanks,
Marcus
comment:6 Changed 4 years ago by ros
Hi Marcus,
I did talk to the Met Office, and they were going to check that functionality - I've yet to hear back. They did say that file is used by the cylc to restart and other things so it may be that it can't be archived because of this.
Regards,
Ros.
comment:7 Changed 4 years ago by ros
- Resolution set to answered
- Status changed from accepted to closed
Hi Marcus,
I'll answer each point in turn..
Hope that helps.
Regards,
Ros