The Reproducibility Project: A Model of Large-Scale Collaboration for Empirical Research on Reproducibility

Abstract

The goal of science is to accumulate knowledge that answers questions such as “How do things work?” and “Why do they work that way?” Scientists use a variety of methodologies to describe, predict, and explain natural phenomena. These methods are so diverse that it is difficult to define a unique scientific method, although all scientific methodologies share the assumption of repeatability (Hempel & Oppenheim, 1948; Kuhn, 1962; Popper, 1934; Salmon, 1989). In this chapter, we first briefly review why replications are highly valued but rarely published. Then we describe a collaborative effort — the Reproducibility Project — to estimate the rate and predictors of reproducibility in psychological science. Finally, we detail how we are conducting this project as a large-scale, distributed, open collaboration. A description of the procedures and challenges may assist and inspire other teams to conduct similar projects in other areas of science.

Keywords: Open Science, collaboration, reproducibility, replication, methodology