Improving parallel efficiency of a Maxwell solver for optical simulation of thin-film solar cells
During the project we optimized the data exchange patterns of a purely MPI parallelized solver for Maxwell's equations that is used to simulate optical properties of thin-film solar cells. Before this project was started the parallel efficiency for job sizes above 64 compute nodes dropped below 65%. In order to improve upon this, during the KONWIHR project, in collaboration with the research group for High Performance Computing of Prof. Dr. Gerhard Wellein, two main optimizations where implemented and tested:
- The first optimization focuses on reducing the communication distance between neighboring processes. By placing processes of one compute node close together within the simulation grid fast intra-node communication can be exploited and slower inter-node communication can be reduced. Since built-in implementations for process reordering of the MPI-library did not show any major advantage a custom process ordering was implemented. By this the parallel efficiency could be raised to 70-75% above 64 compute nodes.
- The second optimization technique utilizes the fact that the six coupled equations of the solver -- three for the electric and three for the magnetic field vectors -- can be ordered in a way that data communication can be overlapped with computations. For this, hybrid OpenMP-MPI parallelization was implemented to use one OpenMP thread for communication and the others for computation. Several different ways to overlap the communication with computation were tested. By those overlapping communication patterns another increase of parallel efficiency by 15-20% could be achieved.
Putting together all implemented optimizations the parallel efficiency of the Maxwell solver could be increased to 90-95% for jobs utilizing more than 64 nodes and the performance for the whole solver could be increased by a factor of 1.5.
- KONWIHR funding: three months during Multicore-Software-Initiative 2014/2015
- Prof. Dr. C. Pflaum, LS Informatik 10 (Systemsimulation), FAU