A Framework for Evaluating Reproducibility and Replicability in Economics

Abstract

A fundamental question to the scientific enterprise is to what extent published scientific findings are credible. This question is related to the reproducibility and replicability of scientific findings where reproducibility is defined as testing if the results of an original study can be reproduced using the same data and replicability is defined as testing if the results of an original study hold in new data. We provide a framework for evaluating reproducibility and replicability in economics and divide reproducibility and replicability studies into five types: computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability and conceptual replicability, and we propose indicators to be reported for each type.

Keywords: reproducibility, replicability, computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability, conceptual replicability

JEL Classification: C18, C19