Parallelization Overhead and HPC-Efficient Programming

Abstract

Easier access to high-performance computing (HPC) systems has enabled parallel computing to become more ubiquitous in numerically solving dynamic economic models. Nevertheless, with little understanding of computer architectures and parallel computing algorithms, executing a program with a large number of cores could result in a significant slowdown rather than a speedup. This study exploits new measure—parallelization overhead—to explicitly quantify the size of the inefficiency that predominantly arises from communication overhead and load imbalances in parallel computing. Further, we demonstrate three programming techniques for an HPC-efficient parallel program that effectively reduces the parallelization overhead.

Keywords: parallel computing, overhead, high-performance computing, lifecycle model

JEL Classification: C63, E37