Benefits of Open and High-Powered Research Outweigh Costs

Journal of Personality and Social Psychology, Forthcoming

41 Pages Posted: 11 Feb 2017  

Etienne P. LeBel

University of Western Ontario

Lorne Campbell

University of Western Ontario

Timothy J. Loving

University of Texas at Austin

Date Written: January 27, 2016

Abstract

Several researchers recently outlined unacknowledged costs of open science practices, arguing these costs may outweigh benefits and stifle discovery of novel findings. We scrutinize these researchers' (1) statistical concern that heightened stringency with respect to false-positives will increase false-negatives and (2) meta-scientific concern that larger samples and executing direct replications engender opportunity costs that will decrease the rate of making novel discoveries. We argue their statistical concern is unwarranted given open science proponents recommend such practices to reduce the inflated Type I error rate from .35 down to .05 and simultaneously call for high-powered research to reduce the inflated Type II error rate. Regarding their meta-concern, we demonstrate that incurring some costs is required to increase the rate (and frequency) of making true discoveries because distinguishing true from false hypotheses requires a low Type I error rate, high statistical power, and independent direct replications. We also examine pragmatic concerns raised regarding adopting open science practices for relationship science (pre-registration, open materials, open data, direct replications, sample size); while acknowledging these concerns, we argue they are overstated given available solutions. We conclude benefits of open science practices outweigh costs for both individual researchers and the collective field in the long run, but that short term costs may exist for researchers because of the currently dysfunctional academic incentive structure. Our analysis implies our field's incentive structure needs to change whereby better alignment exists between researcher's career interests and the field's cumulative progress. We delineate recent proposals aimed at such incentive structure re-alignment.

Keywords: open science practices, independent replication, cumulative knowledge, analytic and design flexibility

Suggested Citation

LeBel, Etienne P. and Campbell, Lorne and Loving, Timothy J., Benefits of Open and High-Powered Research Outweigh Costs (January 27, 2016). Journal of Personality and Social Psychology, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2616384 or http://dx.doi.org/10.2139/ssrn.2616384

Etienne P. LeBel (Contact Author)

University of Western Ontario ( email )

Department of Psychology
SSC
London, Ontario N6A 5C2
Canada

HOME PAGE: http://etiennelebel.com

Lorne Campbell

University of Western Ontario ( email )

1151 Richmond Street
Suite 2
London, Ontario N6A 5B8
Canada

Timothy J. Loving

University of Texas at Austin ( email )

Austin, TX 78712
United States

Paper statistics

Downloads
426
Rank
52,779
Abstract Views
2,837