Replicator Degrees of Freedom Allow Publication of Misleading 'Failures to Replicate'

Abstract

In recent years, the field of psychology has begun to conduct replication tests on a large scale. Here, we show that “replicator degrees of freedom” make it far too easy to obtain and publish false-negative replication results, even while appearing to adhere to strict methodological standards. Specifically, using data from an ongoing debate, we show that commonly-exercised flexibility at the experimental design and data analysis stages of replication testing can make it appear that a finding was not replicated when, in fact, it was. The debate we focus on is representative, on key dimensions, of a large number of other replication tests in psychology that have been published in recent years, suggesting that the lessons of this analysis may be far-reaching. The problems with current practice in replication science that we uncover here are particularly worrisome because they are not adequately addressed by the field’s standard remedies, including pre-registration. Implications for how the field could develop more effective methodological standards for replications are discussed.

Keywords: replication crisis, reproducibility, p-hacking, questionable research practices, researcher degrees of freedom, null hacking