p-Hacking and False Discovery in A/B Testing

44 Pages Posted: 18 Jul 2018  

Ron Berman

University of Pennsylvania - The Wharton School

Leonid Pekelis

OpenDoor

Aisling Scott

Independent

Christophe Van den Bulte

University of Pennsylvania - Marketing Department

Date Written: June 28, 2018

Abstract

We investigate whether online A/B experimenters "p-hack" by stopping their experiments based on the p-value of the treatment effect. Our data contains 2,101 commercial experiments in which experimenters can track the magnitude and significance level of the effect every day of the experiment. We use a regression discontinuity design to detect p-hacking, i.e., the causal effect of reaching a particular p-value on stopping behavior.

Experimenters indeed p-hack, especially for positive effects. Specifically, about 57% of experimenters p-hack when the experiment reaches 90% confidence. Furthermore, approximately 70% of the effects are truly null, and p-hacking increases the false discovery rate (FDR) from 33% to 42% among experiments p-hacked at 90% confidence. Assuming that false discoveries cause experimenters to stop exploring for more effective treatments, we estimate the expected cost of a false discovery to be a loss of 1.95% in lift, which corresponds to the 76th percentile of observed lifts.

Keywords: A/B testing, p-hacking, false discoveries, false positives, experimentation

JEL Classification: C12, C90, C93, M21, M31

Suggested Citation

Berman, Ron and Pekelis, Leonid and Scott, Aisling and Van den Bulte, Christophe, p-Hacking and False Discovery in A/B Testing (June 28, 2018). Available at SSRN: https://ssrn.com/abstract=3204791 or http://dx.doi.org/10.2139/ssrn.3204791

Ron Berman (Contact Author)

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Leonid Pekelis

OpenDoor ( email )

United States

Aisling Scott

Independent ( email )

No Address Available

Christophe Van den Bulte

University of Pennsylvania - Marketing Department ( email )

700 Jon M. Huntsman Hall
3730 Walnut Street
Philadelphia, PA 19104-6340
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
3,658
rank
2,339
Abstract Views
33,294
PlumX