Download this Paper Open PDF in Browser

When Less is More: Data and Power in Advertising Experiments

35 Pages Posted: 31 Oct 2015 Last revised: 15 Feb 2016

Date Written: December 29, 2015


Yahoo! Research partnered with a nationwide retailer to study the effects of online display advertising on both online and in-store purchases. We use a randomized field experiment on 3 million Yahoo! users who are also past customers of the retailer. We find statistically significant evidence that the retailer ads increase sales 3.6% relative to the control group. We show that control ads boost measurement precision by identifying and removing the half of in-campaign sales data that is unaffected by the ads. Less data gives us 31% more precision in our estimates — equivalent to increasing our sample to 5.3 million users. By contrast, we only improve precision by 5% when we include additional covariate data to reduce the residual variance in our experimental regression. The covariate-adjustment strategy disappoints despite exceptional consumer-level data including demographics, ad exposure levels, and two years’ worth of past purchase history.

Keywords: advertising effectiveness, field experiments, digital advertising

JEL Classification: M37, M31, C93, L86

Suggested Citation

Johnson, Garrett A. and Lewis, Randall A. and Reiley, David, When Less is More: Data and Power in Advertising Experiments (December 29, 2015). Simon Business School Working Paper No. FR 15-31. Available at SSRN: or

Garrett Johnson (Contact Author)

Kellogg School of Management ( email )

Evanston, IL
United States


Randall Lewis

Netflix ( email )

Los Gatos, CA
United States
312-RA-LEWIS (Phone)


David Reiley

Google, Inc. ( email )

1600 Amphitheatre Parkway
Second Floor
Mountain View, CA 94043
United States

Paper statistics

Abstract Views