Changes between Version 3 and Version 4 of Projects/UM-NetCDF/NcTest


Ignore:
Timestamp:
01/08/18 15:16:31 (12 months ago)
Author:
jeff
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Projects/UM-NetCDF/NcTest

    v3 v4  
    22== Test Unified Model data output for high resolution atmosphere model == 
    33 
    4 Model run details 
     4* 10 day run  
     5* N1280L85 (2560x1920x85) resolution 
     6* 72x50 processor decomposition 
     7* 2 OpenMP threads per core 
    58 
    6 {{{ 
    7 10 day run, N1280L85 (2560x1920x85) resolution, 72x50 processor decomposition, 2 OpenMP threads per core 
    8 }}} 
     9Results of using different model output formats/data packing 
    910 
    1011||= Wallclock time (hh:mm) =||= Total output size (Gbytes) =||= Size of p9 file (Gbytes) =||= Number of IO server processors =||= Model output =||= Notes =|| 
     
    1213||  09:20  ||  1314  ||  234  ||   0  ||NetCDF output 32 bit  || || 
    1314||  10:03  ||  2630  ||  468  ||   0  ||NetCDF output 64 bit  || || 
    14 ||  20:05  ||   689  ||  126  ||   0  ||NetCDF output 32 bit, 'New Climate' packing profile, deflate level 1  || Time and total file size estimated from 2 day run || 
     15||  20:05  ||   689  ||  126  ||   0  ||NetCDF output 32 bit, 'New Climate' packing profile, deflate level 1  || Time and total output size estimated from 2 day run || 
     16 
     17As can be seen from the above results using data compression with netCDF output is very expensive and its use cannot be recommended. Using 32 bit netCDF file output produces file sizes comparable to the packed fieldfiles output but takes an extra hour to run. This could be worthwhile using if netCDF output is the ultimate goal as an extra post-processing step is no longer necessary. Adding netCDF output functionally to the UM IO servers should make using netCDF output more competitive. 
     18 
     19To better understand how using netCDF compression affects file write time an example test program was written. This program takes a single netCDF variable and writes out various netCDF files using different compression settings. For the results given below a U component of wind on pressure levels field was used, the field size was 2560x1920x3x10, i.e. 3 pressure levels and 10 time steps. 
    1520 
    1621[[Image(figure_1.png,align=center)]] 
     22 
     23It can be seen from this graph that using deflate level 9 (The highest compression level) results in very large file write times and would never be used in practise, the graph below removes these data points to better emphasis the remaining data. 
    1724 
    1825[[Image(figure_2.png,align=center)]] 
    1926 
    2027[[Image(figure_3.png,align=center)]] 
     28 
     29This graph shows that the various deflate levels have a very small affect on file size and therefore it is recommended to always use deflate level 1 which always produces the fastest compression time.