Opened 2 months ago

Closed 3 weeks ago

#3338 closed defect (answered)

different sizes for dump file on ARCHER

Reported by: pmcguire Owned by: um_support
Component: Data Keywords: postproc, dump file size, ARCHER
Cc: Platform: ARCHER
UM Version: 11.5

Description (last modified by pmcguire)

Hi CMS Helpdesk
My postproc app was repeatedly failing, even though I increased the wall clock time.
The last file that it was trying to copy/postproc from the cylc-run directory to the archive directory was this dump file, which reports 1.237 TB as its file size with ls, but only 0.0088TB with du. The other dump files are more normal, at 0.008TB=8GB. I moved this file just now from the History_data directory to its parent directory, so that maybe postproc will ignore it.

Any idea why there is a different size reported? Is it corrupted somehow? I guess I can live without this dump file? Or should I try to archive it somewhere? Or should I immediately delete the file, to save space on ARCHER, if it's really taking 1.237 TB?
Patrick


pmcguire@eslogin003:~> ls -ltr /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/bv932a.da20080501_00

-rw-r—r— 1 pmcguire n02 1237172883456 Aug 4 17:45 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/bv932a.da20080501_00


pmcguire@eslogin003:~> du -scm /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/bv932a.da20080501_00

8862 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/bv932a.da20080501_00

8862 total


pmcguire@eslogin003:~> ls -ltr /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da2008*

-rw-r—r— 1 pmcguire n02 8291094528 Aug 2 05:59 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20080301_00

-rw-r—r— 1 pmcguire n02 8291094528 Aug 2 12:01 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20080401_00

-rw-r—r— 1 pmcguire n02 8291094528 Aug 4 18:37 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20080601_00

-rw-r—r— 1 pmcguire n02 8291094528 Aug 4 19:30 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20080701_00

-rw-r—r— 1 pmcguire n02 8291094528 Aug 4 20:28 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20080801_00

-rw-r—r— 1 pmcguire n02 8291094528 Aug 4 21:24 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20080901_00

-rw-r—r— 1 pmcguire n02 8291094528 Aug 4 22:19 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20081001_00

-rw-r—r— 1 pmcguire n02 8291094528 Aug 4 23:12 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20081101_00

-rw-r—r— 1 pmcguire n02 8291094528 Aug 5 00:04 /work/n02/n02/pmcguire/cylc-run/u-bv932b/share/data/History_Data/bv932a.da20081201_00

Change History (5)

comment:1 Changed 2 months ago by pmcguire

  • Description modified (diff)

comment:2 Changed 2 months ago by pmcguire

  • Description modified (diff)

comment:3 Changed 2 months ago by pmcguire

  • Description modified (diff)

comment:4 Changed 2 months ago by ros

Hi Patrick,

Had a very quick look. Not sure I've seen a difference quite like that on ARCHER before. Have seen similar on JASMIN.

Running du with the --apparent-size option looks to indicate there's internal fragmentation (or indirect blocks) of that file which can lead to a big difference between disk usage and actual file size….

ARCHER-xc30> du -sh bv932a.da20080501_00 
8.7G	bv932a.da20080501_00

ARCHER-xc30> du -sh bv932a.da20080501_00 --apparent-size
1.2T	bv932a.da20080501_00

Not sure if that helps you or not.
Cheers,
Ros.

Last edited 2 months ago by ros (previous) (diff)

comment:5 Changed 3 weeks ago by ros

  • Resolution set to answered
  • Status changed from new to closed

Closed due to lack of activity

Note: See TracTickets for help on using tickets.