The Unfavorable Economics of Measuring the Returns to Advertising

Randall A. Lewis


Justin M. Rao

Microsoft Research; Microsoft Corporation - Microsoft Research - Redmond

September 18, 2014

Twenty-five large field experiments with major U.S. retailers and brokerages, each reaching millions of customers and collectively representing $2.8 million in advertising expenditure, reveal that measuring the returns to advertising is exceedingly difficult. The median confidence interval on ROI is over 100% wide, the smallest exceeds 50%. Detailed sales data show that, relative to the per capita cost of the advertising, individual-level sales are incredibly volatile; a coefficient of variation of 10 is common. Hence, informative advertising experiments can easily require more than ten million person-weeks, making experiments costly and potentially infeasible for many firms. Despite these unfavorable economics, randomized control trials represent progress by injecting new, unbiased information into the market. The statistically small impact of profitable advertising amid such noise means that selection bias is a crippling concern for widely-employed observational methods. We discuss how these biases and weak informational feedback from experiments fundamentally impact both advertisers and publishers.

Number of Pages in PDF File: 35

Keywords: advertising, field experiments, causal inference, electronic commerce, return on investment, information

JEL Classification: L10, M37, C93

Open PDF in Browser Download This Paper

Date posted: December 14, 2013 ; Last revised: September 19, 2014

Suggested Citation

Lewis, Randall A. and Rao, Justin M., The Unfavorable Economics of Measuring the Returns to Advertising (September 18, 2014). Available at SSRN: https://ssrn.com/abstract=2367103 or http://dx.doi.org/10.2139/ssrn.2367103

Contact Information

Randall A. Lewis
Netflix ( email )
Los Gatos, CA
United States
312-RA-LEWIS (Phone)
HOME PAGE: http://www.econinformatics.com/
Justin M. Rao (Contact Author)
Microsoft Research ( email )
641 Avenue of Americas
7th Floor
New York, NY 11249
United States
Microsoft Corporation - Microsoft Research - Redmond ( email )
Building 99
Redmond, WA
United States
Feedback to SSRN

Paper statistics
Abstract Views: 12,220
Downloads: 2,511
Download Rank: 3,524