wiki:Projects/UM-NetCDF/NcTest

Test Unified Model data output for high resolution atmosphere model

  • 10 day run
  • N1280L85 (2560x1920x85) resolution
  • 72x50 processor decomposition
  • 2 OpenMP threads per core

Results of using different model output formats/data packing

Wallclock time (hh:mm) Total output size (Gbytes) Size of p9 file (Gbytes) Number of IO server processors Model output Notes
08:18 1325 232 36 Fieldfiles output, 'New Climate' packing profile
09:20 1314 234 0 NetCDF output 32 bit
10:03 2630 468 0 NetCDF output 64 bit
20:05 689 126 0 NetCDF output 32 bit, 'New Climate' packing profile, deflate level 1 Time and total output size estimated from 2 day run

As can be seen from the above results using data compression with netCDF output is very expensive and its use cannot be recommended. Using 32 bit netCDF file output produces file sizes comparable to the packed fieldfiles output but takes an extra hour to run. This could be worthwhile using if netCDF output is the ultimate goal as an extra post-processing step is no longer necessary. Adding netCDF output functionally to the UM IO servers should make using netCDF output more competitive.

To better understand how using netCDF compression affects file write time an example test program was written. This program takes a single netCDF variable and writes out various netCDF files using different compression settings. For the results given below a U component of wind on pressure levels field was used, the field size was 2560x1920x3x10, i.e. 3 pressure levels and 10 time steps.

Wallclock time writing netCDF file using various compression values

It can be seen from this graph that using deflate level 9 (The highest compression level) results in very large file write times and would never be used in practise, the graph below removes these data points to better emphasis the remaining data.

Wallclock time writing netCDF file using various compression values - highest compression level removed

Size of netCDF file using various compression values

This graph shows that the various deflate levels have a very small affect on file size and therefore it is recommended to always use deflate level 1 which will has the fastest compression time.

Last modified 16 months ago Last modified on 01/08/18 15:22:57

Attachments (3)

Download all attachments as: .zip