When Less is More: Data and Power in Advertising Experiments

35 Pages Posted: 31 Oct 2015 Last revised: 15 Feb 2016

See all articles by Garrett Johnson

Garrett Johnson

Questrom School of Business

Randall A. Lewis


David Reiley

Pandora Media, Inc.; UC Berkeley School of Information

Date Written: December 29, 2015


Yahoo! Research partnered with a nationwide retailer to study the effects of online display advertising on both online and in-store purchases. We use a randomized field experiment on 3 million Yahoo! users who are also past customers of the retailer. We find statistically significant evidence that the retailer ads increase sales 3.6% relative to the control group. We show that control ads boost measurement precision by identifying and removing the half of in-campaign sales data that is unaffected by the ads. Less data gives us 31% more precision in our estimates — equivalent to increasing our sample to 5.3 million users. By contrast, we only improve precision by 5% when we include additional covariate data to reduce the residual variance in our experimental regression. The covariate-adjustment strategy disappoints despite exceptional consumer-level data including demographics, ad exposure levels, and two years’ worth of past purchase history.

Keywords: advertising effectiveness, field experiments, digital advertising

JEL Classification: M37, M31, C93, L86

Suggested Citation

Johnson, Garrett and Lewis, Randall A. and Reiley, David H., When Less is More: Data and Power in Advertising Experiments (December 29, 2015). Simon Business School Working Paper No. FR 15-31, Available at SSRN: https://ssrn.com/abstract=2683621 or http://dx.doi.org/10.2139/ssrn.2683621

Garrett Johnson (Contact Author)

Questrom School of Business ( email )

595 Commonwealth Avenue
Boston, MA 02215
United States
6173534677 (Phone)

HOME PAGE: http://www.bu.edu/questrom/faculty-research/faculty-directory/garrett-johnson/

Randall A. Lewis

Amazon ( email )

312-RA-LEWIS (Phone)

David H. Reiley

Pandora Media, Inc. ( email )

Oakland, CA 94612
United States

UC Berkeley School of Information ( email )

102 South Hall
Berkeley, CA 94720-4600
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
PlumX Metrics