Changes between Version 5 and Version 6 of Projects/OpenIFS-IO

23/09/13 15:19:16 (8 years ago)


  • Projects/OpenIFS-IO

    v5 v6  
    1818Considerable progress has been made with this project. The OpenIFS repository has been created on PUMA and the OpenIFS code installed. Mark has installed the necessary grib libraries on HECToR and the FDB libraries under a package account for maintainability and has set up environment modules to help manage the software installation. The FDB calls have been activated in the OpenIFS code and we have successfully extracted and built the model on HECToR. Glenn has made available several models at differing resolutions for testing. We have concentrated on the t1279 model (about 25km global resolution). Several scenarios have been investigated - the results are summarized in the figure below. A baseline performance with minimal IO is established for 1 and 8 OMP threads the model scales well, especially when run with 8 OMP threads (see figure below right). OpenIFS running with is 'single-writer' output scheme shows the characteristic performance slow down as a single processor throttles the rest in order to gather and output data. Our test generates hourly output for a total of 58GB/model day; time spent performing output is a very significant portion of the total (~25% of wallclock for 1024 processors, rising to ~40% wallclock for 4096 processors). Running with the FDB scheme shows an impressive performance improvement, whereby the time taken for asynchronous multiprocessor output is virtually hidden and amounts to not more than 5% of the wallclock time for the run. 
    20 [[Image(oifs-t1279.png,left)]] 
     31=== How to get on IOFS job running on HECToR === 
     33There are a couple of preliminary administrative tasks which need to be taken care of first.  
     35* Your institution will need an OpenIFS license - check with Glenn Carver (ECMWF) 
     36* You will need an account on HECToR 
     37* You will need to be a member of the oifs package account on HECToR. Contact CMS to arrange to be added to this group.  
     38* You will need to have access to the OpenIFS code repository on PUMA - contact CMS 
     40It will be helpful if you have some knowledge of the Flexible Configuration Management (FCM) system. We use FCM to manage the OpenIFS code and in addition to manage the OpenIFS build. Much of the work has been done which will enable you to get started on HECToR, but it will benefit you and make your work more efficient if you become familiar with some simple FCM operations, such as commit, check-out, add, revert, branch-create. FCM is described in depth in complete online documentation at and an FCM tutorial is available at  
     43=== Check out the OIFS code === 
     45You will need to set up the correct environment to build and run OIFS jobs. Since HECToR has different hardware on its compute and service nodes, there is an extra consideration when using some OIFS-related utilities on the service nodes (more later). Let's assume that you are interested in building and running the model only for now. At the HECToR command line, type 
     48module use ~oifs/modules 
     49module load openifs_cce/0.0.2 
     52Loading the openifs module sets some paths required for the build and run and has some checking to ensure that there is no conflict with currently loaded modules. Typing 
     55module show openifs_cce/0.0.2 
     58gives details of what the module does, namely 
     61module-whatis    CCE versions of support libraires to build OpenIFS  
     62conflict         PrgEnv-gnu  
     63conflict         PrgEnv-pgi  
     64prereq   cce/8.1.8  
     65module           load grib_api_cce/0.0.2  
     66module           load fdb_cce/0.0.2  
     67prepend-path     PATH /work/n02/n02/hum/fcm/bin  
     68setenv           OIFS_IFSDATA /work/y07/y07/oifs/data/ecmwf/ifsdata  
     69setenv           OIFS_ARCH x86_64  
     70setenv           OIFS_COMP cce_fdb  
     71setenv           OIFS_BUILD opt  
     74If your normal module set up is with the gnu or pgi programming environments, then an attempt to load openifs_cce/0.0.2 will fail, and it will also fail unless you have the Cray cce1/8.1.8 compiler loaded. Use '''module swap''' to ensure an appropriate starting module configuration. 
     77On HECToR create a suitably named directory in your /home space. We recommend building in /home which handles small files more efficiently than does /work for a quicker build. Now check out the OIFS code tree with '''fcm co'''. Here is an example 
     80fcm co fcm:oifs/branches/dev/grenville/38r1v02_fdb_updates@16 
     83This cumbersome command can be made simpler by use of FCM keywords and of course you can use aliasing to reduce the amount of typing required. However, it is instructive to see the command in its entirety (actually we have already defined the keyword oifs to save some typing). The text after fcm co is the URL for the code. In this case the code is in a '''branch''' owned by '''dev'''eloper '''grenville'''; the breanch is called '''38r1v02_fdb_updates''' and the code is at ('''@''') revision number '''16'''