Latent Stratification for Incrementality Experiments

52 Pages Posted: 19 Mar 2018 Last revised: 10 May 2023

See all articles by Ron Berman

Ron Berman

University of Pennsylvania - The Wharton School

Elea McDonnell Feit

Drexel University - Department of Marketing

Date Written: May 9, 2023

Abstract

Incrementality experiments compare customers exposed to a marketing action designed to increase sales to those randomly assigned to a control group. These experiments suffer from noisy responses which make precise estimation of the average treatment effect (ATE) and marketing ROI difficult. We develop a model that improves the precision by estimating separate treatment effects for three latent strata defined by potential outcomes in the experiment -- customers who would buy regardless of ad exposure, those who would buy only if exposed to ads and those who would not buy regardless. The overall ATE is estimated by averaging the strata-level effects, and this produces a more precise estimator of the ATE over a wide range of conditions typical of marketing experiments. Analytical results and simulations show that the method decreases the sampling variance of the ATE most when (1) there are large differences in the treatment effect between latent strata and (2) the model used to estimate the strata-level effects is well-identified. Applying the procedure to 5 catalog experiments shows a reduction of 30-60% in the variance of the overall ATE. This leads to a substantial decrease in decision errors when the estimator is used to determine whether ads should be continued or discontinued.

Keywords: Advertising, incrementality experiments, lift testing, A/B testing, holdout experiments, average treatment effect, principal stratification, causal inference

JEL Classification: C12, C93

Suggested Citation

Berman, Ron and Feit, Elea McDonnell, Latent Stratification for Incrementality Experiments (May 9, 2023). The Wharton School Research Paper, Wharton Customer Analytics Initiative Research Paper, Available at SSRN: https://ssrn.com/abstract=3140631 or http://dx.doi.org/10.2139/ssrn.3140631

Ron Berman

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Elea McDonnell Feit (Contact Author)

Drexel University - Department of Marketing ( email )

United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
863
Abstract Views
4,392
Rank
50,930
PlumX Metrics