Principal Stratification for Advertising Experiments

Abstract

Advertising experiments often suffer from noisy responses making precise estimation of the average treatment effect (ATE) and evaluating ROI difficult. We develop a principal stratification model that improves the precision of the ATE by dividing the customers into three strata -- those who buy regardless of ad exposure, those who buy only if exposed to ads and those who do not buy regardless. The method decreases the variance of the ATE by separating out the typically large share of customers who never buy and therefore have individual treatment effects that are exactly zero. Applying the procedure to 5 catalog mailing experiments with sample sizes around 140,000 shows a reduction of 36-57% in the variance of the estimate. When we include pre-randomization covariates that predict stratum membership, we find that estimates of customers' past response to similar advertising are a good predictor of stratum membership, even if such estimates are biased because past advertising was targeted. Customers who have not purchased recently are also more likely to be in the "never purchase" stratum. We provide simple summary statistics that firms can compute from their own experiment data to determine if the procedure is expected to be beneficial before applying it.

Keywords: advertising, incrementality, lift testing, holdout experiments, average treatment effect, principal stratification, causal inference