A Framework for Evaluating Reproducibility and Replicability in Economics

Abstract

We propose a framework for evaluating reproducibility and replicability in economics. Reproducibility is defined as testing if the results of an original study can be reproduced using the same data and replicability is defined as testing if the results of an original study hold in new data. We further divide reproducibility and replicability studies into five types: computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability and conceptual replicability. In addition to this typology we propose indicators to measure the degree of reproducibility and replicability in both individual studies and for a group of studies.

Keywords: reproducibility, replicability, computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability, conceptual replicability

JEL Classification: C18, C19