A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook

Forthcoming at Marketing Science

54 Pages Posted: 9 Sep 2017 Last revised: 7 Oct 2018

Date Written: September 23, 2018

Abstract

Measuring the causal effects of digital advertising remains challenging despite the availability of granular data. Unobservable factors make exposure endogenous, and advertising’s effect on outcomes tends to be small. In principle, these concerns could be addressed using randomized controlled trials (RCTs). In practice, few online ad campaigns rely on RCTs, and instead use observational methods to estimate ad effects. We assess empirically whether the variation in data typically available in the advertising industry enables observational methods to recover the causal effects of online advertising. Using data from 15 US advertising experiments at Facebook comprising 500 million user-experiment observations and 1.6 billion ad impressions, we contrast the experimental results to those obtained from multiple observational models. The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables. In our setting, advances in causal inference methods do not allow us to isolate the exogenous variation needed to estimate the treatment effects. We also characterize the incremental explanatory power our data would require to enable observational methods to successfully measure advertising effects. Our findings suggest that commonly used observational approaches based on the data usually available in the industry often fail to accurately measure the true effect of advertising.

Keywords: field experiments, causal inference, advertising effects, digital advertising, observational methods

JEL Classification: L10, M37, C93

Suggested Citation

Gordon, Brett R. and Zettelmeyer, Florian and Bhargava, Neha and Chapsky, Dan, A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook (September 23, 2018). Forthcoming at Marketing Science. Available at SSRN: https://ssrn.com/abstract=3033144 or http://dx.doi.org/10.2139/ssrn.3033144

Brett R. Gordon (Contact Author)

Northwestern University - Kellogg School of Management ( email )

2001 Sheridan Road
Evanston, IL 60208
United States

Florian Zettelmeyer

Northwestern University - Kellogg School of Management ( email )

2001 Sheridan Road
Evanston, IL 60208
United States

Neha Bhargava

Facebook ( email )

1601 S. California Ave.
Palo Alto, CA 94304
United States

Dan Chapsky

Facebook ( email )

1601 S. California Ave.
Palo Alto, CA 94304
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
1,063
rank
18,736
Abstract Views
3,339
PlumX Metrics