Download this Paper Open PDF in Browser

Better P-Curves: Making P-Curve Analysis More Robust to Errors, Fraud, and Ambitious P-Hacking, A Reply to Ulrich and Miller

Simonsohn, Uri, Joseph P. Simmons, and Leif D. Nelson (2015), “Better P-Curves: Making P-Curve Analysis More Robust To Errors, Fraud, and Ambitious P-Hacking, A Reply To Ulrich and Miller (2015),” Journal of Experimental Psychology: General, 144 (December), 1146-1152

7 Pages Posted: 24 Aug 2015 Last revised: 16 Apr 2016

Uri Simonsohn

University of Pennsylvania - The Wharton School

Joseph P. Simmons

University of Pennsylvania - The Wharton School

Leif D. Nelson

University of California, Berkeley - Haas School of Business

Date Written: July 10, 2015

Abstract

When studies examine true effects, they generate right-skewed p-curves, distributions of statistically significant results with more low (.01s) than high (.04s) p-values. What else can cause a right-skewed p-curve? First, we consider the possibility that researchers report only the smallest significant p-value (as conjectured by Ulrich & Miller, 2015), concluding that it is a very uncommon problem. We then consider more common problems, including (1) p-curvers selecting the wrong p-values, (2) fake data, (3) honest errors, and (4) ambitiously p-hacked (beyond p<.05) results. We evaluate the impact of these common problems on the validity of p-curve analysis, and provide practical solutions that substantially increase its robustness.

Keywords: publication bias, selective reporting, p-hacking, false-positive psychology, hypothesis testing, meta-analysis

Suggested Citation

Simonsohn, Uri and Simmons, Joseph P. and Nelson, Leif D., Better P-Curves: Making P-Curve Analysis More Robust to Errors, Fraud, and Ambitious P-Hacking, A Reply to Ulrich and Miller (July 10, 2015). Simonsohn, Uri, Joseph P. Simmons, and Leif D. Nelson (2015), “Better P-Curves: Making P-Curve Analysis More Robust To Errors, Fraud, and Ambitious P-Hacking, A Reply To Ulrich and Miller (2015),” Journal of Experimental Psychology: General, 144 (December), 1146-1152. Available at SSRN: https://ssrn.com/abstract=2649230

Uri Simonsohn (Contact Author)

University of Pennsylvania - The Wharton School ( email )

3730 Walnut Street
JMHH 500
Philadelphia, PA 19104-6365
United States

Joseph Simmons

University of Pennsylvania - The Wharton School ( email )

3733 Spruce Street
Philadelphia, PA 19104-6374
United States

Leif Nelson

University of California, Berkeley - Haas School of Business ( email )

545 Student Services Building, #1900
2220 Piedmont Avenue
Berkeley, CA 94720
United States

Paper statistics

Downloads
1,308
Rank
11,805
Abstract Views
4,665