The Unfavorable Economics of Measuring the Returns to Advertising

Abstract

Twenty-five large field experiments with major U.S. retailers and brokerages, each reaching millions of customers and collectively representing $2.8 million in advertising expenditure, reveal that measuring the returns to advertising is exceedingly difficult. The median confidence interval on ROI is over 100% wide, the smallest exceeds 50%. Detailed sales data show that, relative to the per capita cost of the advertising, individual-level sales are incredibly volatile; a coefficient of variation of 10 is common. Hence, informative advertising experiments can easily require more than ten million person-weeks, making experiments costly and potentially infeasible for many firms. Despite these unfavorable economics, randomized control trials represent progress by injecting new, unbiased information into the market. The statistically small impact of profitable advertising amid such noise means that selection bias is a crippling concern for widely-employed observational methods. We discuss how these biases and weak informational feedback from experiments fundamentally impact both advertisers and publishers.

Keywords: advertising, field experiments, causal inference, electronic commerce, return on investment, information

JEL Classification: L10, M37, C93