Better P-Curves: Making P-Curve Analysis More Robust to Errors, Fraud, and Ambitious P-Hacking, A Reply to Ulrich and Miller

Abstract

When studies examine true effects, they generate right-skewed p-curves, distributions of statistically significant results with more low (.01s) than high (.04s) p-values. What else can cause a right-skewed p-curve? First, we consider the possibility that researchers report only the smallest significant p-value (as conjectured by Ulrich & Miller, 2015), concluding that it is a very uncommon problem. We then consider more common problems, including (1) p-curvers selecting the wrong p-values, (2) fake data, (3) honest errors, and (4) ambitiously p-hacked (beyond p<.05) results. We evaluate the impact of these common problems on the validity of p-curve analysis, and provide practical solutions that substantially increase its robustness.

Keywords: publication bias, selective reporting, p-hacking, false-positive psychology, hypothesis testing, meta-analysis