A Hybrid MPI–OpenMP Parallel Algorithm and Performance Analysis for an Ensemble Square Root Filter Designed for Multiscale Observations

• Published in: Journal of atmospheric and oceanic technology Vol. 30; no. 7; pp. 1382 - 1397
• Main Authors: Wang, Yunheng, Jung, Youngsun, Supinie, Timothy A., Xue, Ming
• Format: Journal Article
• Language: English
• Published: Boston American Meteorological Society 01.07.2013
• Subjects: Algorithms; Analysis; Communication; Communication satellites; Computer memory; Configurations; Data assimilation; Data collection; Decomposition; Distributed shared memory; Domain decomposition; Localization; Marine; Message passing; Multiprocessing; Multiprocessing (computers); Nodes; Parallel programming; Radar; Radar data; Radar systems; Roots; Satellite observation; Supercomputers; Topology; Variables; Weather forecasting
• ISSN: 0739-0572; 1520-0426
• DOI: 10.1175/JTECH-D-12-00165.1

A hybrid parallel scheme for the ensemble square root filter (EnSRF) suitable for parallel assimilation of multiscale observations, including those from dense observational networks such as those of radar, is developed based on the domain decomposition strategy. The scheme handles internode communication through a message passing interface (MPI) and the communication within shared-memory nodes via Open Multiprocessing (OpenMP) threads. It also supports pure MPI and pure OpenMP modes. The parallel framework can accommodate high-volume remote-sensed radar (or satellite) observations as well as conventional observations that usually have larger covariance localization radii. The performance of the parallel algorithm has been tested with simulated and real radar data. The parallel program shows good scalability in pure MPI and hybrid MPI–OpenMP modes, while pure OpenMP runs exhibit limited scalability on a symmetric shared-memory system. It is found that in MPI mode, better parallel performance is achieved with domain decomposition configurations in which the leading dimension of the state variable arrays is larger, because this configuration allows for more efficient memory access. Given a fixed amount of computing resources, the hybrid parallel mode is preferred to pure MPI mode on supercomputers with nodes containing shared-memory cores. The overall performance is also affected by factors such as the cache size, memory bandwidth, and the networking topology. Tests with a real data case with a large number of radars confirm that the parallel data assimilation can be done on a multicore supercomputer with a significant speedup compared to the serial data assimilation algorithm.