Principal Stratification for Advertising Experiments

38 Pages Posted: 19 Mar 2018 Last revised: 20 Nov 2019

See all articles by Ron Berman

Ron Berman

University of Pennsylvania, The Wharton School

Elea McDonnell Feit

Drexel University - Department of Marketing

Date Written: November 19, 2019

Abstract

Advertising experiments often suffer from noisy responses making precise estimation of the average treatment effect (ATE) and evaluating ROI difficult. We develop a principal stratification model that improves the precision of the ATE by dividing the customers into three strata -- those who buy regardless of ad exposure, those who buy only if exposed to ads and those who do not buy regardless. The method decreases the variance of the ATE by separating out the typically large share of customers who never buy and therefore have individual treatment effects that are exactly zero. Applying the procedure to 5 catalog mailing experiments with sample sizes around 140,000 shows a reduction of 36-57% in the variance of the estimate. When we include pre-randomization covariates that predict stratum membership, we find that estimates of customers' past response to similar advertising are a good predictor of stratum membership, even if such estimates are biased because past advertising was targeted. Customers who have not purchased recently are also more likely to be in the "never purchase" stratum. We provide simple summary statistics that firms can compute from their own experiment data to determine if the procedure is expected to be beneficial before applying it.

Keywords: advertising, incrementality, lift testing, holdout experiments, average treatment effect, principal stratification, causal inference

Suggested Citation

Berman, Ron and Feit, Elea McDonnell, Principal Stratification for Advertising Experiments (November 19, 2019). The Wharton School Research Paper; Wharton Customer Analytics Initiative Research Paper. Available at SSRN: https://ssrn.com/abstract=3140631 or http://dx.doi.org/10.2139/ssrn.3140631

Ron Berman

University of Pennsylvania, The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104
United States

Elea McDonnell Feit (Contact Author)

Drexel University - Department of Marketing ( email )

United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
422
Abstract Views
1,996
rank
71,046
PlumX Metrics