Better P-Curves: Making P-Curve Analysis More Robust to Errors, Fraud, and Ambitious P-Hacking, A Reply to Ulrich and Miller
Simonsohn, Uri, Joseph P. Simmons, and Leif D. Nelson (2015), “Better P-Curves: Making P-Curve Analysis More Robust To Errors, Fraud, and Ambitious P-Hacking, A Reply To Ulrich and Miller (2015),” Journal of Experimental Psychology: General, 144 (December), 1146-1152
7 Pages Posted: 24 Aug 2015 Last revised: 16 Apr 2016
Date Written: July 10, 2015
When studies examine true effects, they generate right-skewed p-curves, distributions of statistically significant results with more low (.01s) than high (.04s) p-values. What else can cause a right-skewed p-curve? First, we consider the possibility that researchers report only the smallest significant p-value (as conjectured by Ulrich & Miller, 2015), concluding that it is a very uncommon problem. We then consider more common problems, including (1) p-curvers selecting the wrong p-values, (2) fake data, (3) honest errors, and (4) ambitiously p-hacked (beyond p<.05) results. We evaluate the impact of these common problems on the validity of p-curve analysis, and provide practical solutions that substantially increase its robustness.
Keywords: publication bias, selective reporting, p-hacking, false-positive psychology, hypothesis testing, meta-analysis
Suggested Citation: Suggested Citation