The Unfavorable Economics of Measuring the Returns to Advertising

35 Pages Posted: 14 Dec 2013 Last revised: 19 Sep 2014

See all articles by Randall A. Lewis

Randall A. Lewis


Justin M. Rao

Microsoft Research; Microsoft Corporation - Microsoft Research - Redmond

Date Written: September 18, 2014


Twenty-five large field experiments with major U.S. retailers and brokerages, each reaching millions of customers and collectively representing $2.8 million in advertising expenditure, reveal that measuring the returns to advertising is exceedingly difficult. The median confidence interval on ROI is over 100% wide, the smallest exceeds 50%. Detailed sales data show that, relative to the per capita cost of the advertising, individual-level sales are incredibly volatile; a coefficient of variation of 10 is common. Hence, informative advertising experiments can easily require more than ten million person-weeks, making experiments costly and potentially infeasible for many firms. Despite these unfavorable economics, randomized control trials represent progress by injecting new, unbiased information into the market. The statistically small impact of profitable advertising amid such noise means that selection bias is a crippling concern for widely-employed observational methods. We discuss how these biases and weak informational feedback from experiments fundamentally impact both advertisers and publishers.

Keywords: advertising, field experiments, causal inference, electronic commerce, return on investment, information

JEL Classification: L10, M37, C93

Suggested Citation

Lewis, Randall A. and Rao, Justin M., The Unfavorable Economics of Measuring the Returns to Advertising (September 18, 2014). Available at SSRN: or

Randall A. Lewis

Amazon ( email )

312-RA-LEWIS (Phone)

Justin M. Rao (Contact Author)

Microsoft Research ( email )

641 Avenue of Americas
7th Floor
New York, NY 11249
United States

Microsoft Corporation - Microsoft Research - Redmond ( email )

Building 99
Redmond, WA
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
PlumX Metrics