Replicator Degrees of Freedom Allow Publication of Misleading 'Failures to Replicate'

56 Pages Posted: 25 Jun 2019 Last revised: 13 Jul 2019

See all articles by Christopher Bryan

Christopher Bryan

University of Chicago, Booth School of Business

David S. Yeager

University of Texas at Austin

Joseph O’Brien

University of Texas at Austin

Date Written: June 21, 2019

Abstract

In recent years, the field of psychology has begun to conduct replication tests on a large scale. Here, we show that “replicator degrees of freedom” make it far too easy to obtain and publish false-negative replication results, even while appearing to adhere to strict methodological standards. Specifically, using data from an ongoing debate, we show that commonly-exercised flexibility at the experimental design and data analysis stages of replication testing can make it appear that a finding was not replicated when, in fact, it was. The debate we focus on is representative, on key dimensions, of a large number of other replication tests in psychology that have been published in recent years, suggesting that the lessons of this analysis may be far-reaching. The problems with current practice in replication science that we uncover here are particularly worrisome because they are not adequately addressed by the field’s standard remedies, including pre-registration. Implications for how the field could develop more effective methodological standards for replications are discussed.

Keywords: replication crisis, reproducibility, p-hacking, questionable research practices, researcher degrees of freedom, null hacking

Suggested Citation

Bryan, Christopher and Yeager, David S. and O’Brien, Joseph, Replicator Degrees of Freedom Allow Publication of Misleading 'Failures to Replicate' (June 21, 2019). Available at SSRN: https://ssrn.com/abstract=3408200 or http://dx.doi.org/10.2139/ssrn.3408200

Christopher Bryan (Contact Author)

University of Chicago, Booth School of Business ( email )

5807 S. Woodlawn Avenue
Chicago, IL 60637
United States

HOME PAGE: http://https://voices.uchicago.edu/bryan/

David S. Yeager

University of Texas at Austin

2317 Speedway
Austin, TX 78712
United States

Joseph O’Brien

University of Texas at Austin

2317 Speedway
Austin, TX 78712
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
741
Abstract Views
4,049
rank
33,040
PlumX Metrics