Climate Model Initialisation and Improvement using Particle Filter Code is a NERC funded project lead by PJ van Leeuwen at Reading University. NCAS-CMS receives funding for Simon Wilson from the project. The aim of the project is to include complex climate models in particle filters, which currently work in conjunction with simple models only. Simon is developing the scheme to use HadCM3 as the particle (the model) and is making further enhancements to the particle filter framework to improve its efficiency in the HPC environment.
The original code had the particle filter (PF) and simple model mixed together; a complete rewrite has been undertaken to separate these and turn the PF into a coupler. This allows the PF to adapted for other codes with minimal changes to both the coupler and model codes. Model codes are likely to be MPI, so careful consideration is needed for the communicators.
The PF and models are launched with single mpiexec or aprun, and so share a communicator. We will have an ensemble of HadCM3 instances running separately and undertaking non-blocking communication with the particle filter until at the end of each model day, at which point, the main prognostics sent to the PF code are processed and perturbed and returned them to model. All communications are through MPI Send/Receive? pairs.
On the Cray, the PF runs on 1 node and is currently scalar but soon will run parallelized. The PF has one only 1 collective, and is almost ideally parallelizable.
The attached file (pf-communicators.png) shows how the communicators are currently arranged. Each model in the ensemble runs on its processors (pe0, pe1, pe2….) and has its own Model Communicator through which all model intra-communications take place. The PF has its own communicator to ensure that its internal communications are isolated and the Data communicator handles the model - pf communications. All processes reside in the Global communicator.
Simon is working on the project with Phil Browne - they have the system working with the Lorenz 1963 model and a test version has been successfully implemented with the TELEMAC shallow-water model at HR Wallingford.
The Particle Filter intrastructure will be released as a software package "EMPIRE". This will enable any model code to be interfaced with particle filter via simple MPI send and receives of its state vector.
Possible further involvement to use HadGEM3 in place of HadCM3.