False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
Joseph P. Simmons
University of Pennsylvania - The Wharton School; University of Pennsylvania - Operations & Information Management Department
Leif D. Nelson
University of California, Berkeley - Haas School of Business
University of Pennsylvania - The Wharton School
May 23, 2011
Psychological Science, 2011
In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
Number of Pages in PDF File: 8
Keywords: Methodology, Motivated Reasoning, Publication, DisclosureAccepted Paper Series
Date posted: May 24, 2011 ; Last revised: July 5, 2012
© 2014 Social Science Electronic Publishing, Inc. All Rights Reserved.
This page was processed by apollo4 in 0.266 seconds