Latent Stratification for Incrementality Experiments

44 Pages Posted: 19 Mar 2018 Last revised: 8 Aug 2022

See all articles by Ron Berman

Ron Berman

University of Pennsylvania - The Wharton School

Elea McDonnell Feit

Drexel University - Department of Marketing

Date Written: August 5, 2022

Abstract

Incrementality experiments compare customers exposed to a marketing action designed to increase sales to those randomly assigned to a control group. These experiments suffer from noisy responses which make precise estimation of the average treatment effect (ATE) and marketing ROI difficult. We develop a model that improves the precision by estimating separate treatment effects for three latent strata defined by potential outcomes in the experiment -- customers who would buy regardless of ad exposure, those who would buy only if exposed to ads and those who would not buy regardless. The overall ATE is estimated by averaging the strata-level effects, and this produces a more precise estimator of the ATE over a wide range of conditions typical of marketing experiments. Applying the procedure to 5 catalog experiments shows a reduction of 30-60% in the variance of the overall ATE. Analytical results and simulations show that the method decreases the variance of the ATE most when (1) there are large differences in the treatment effect between latent strata and (2) the model is well-identified.

Keywords: advertising, incrementality experiments, lift testing, A/B testing, holdout experiments, average treatment effect, principal stratification, causal inference

JEL Classification: C12, C93

Suggested Citation

Berman, Ron and Feit, Elea McDonnell, Latent Stratification for Incrementality Experiments (August 5, 2022). The Wharton School Research Paper, Wharton Customer Analytics Initiative Research Paper, Available at SSRN: https://ssrn.com/abstract=3140631 or http://dx.doi.org/10.2139/ssrn.3140631

Ron Berman

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Elea McDonnell Feit (Contact Author)

Drexel University - Department of Marketing ( email )

United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
692
Abstract Views
3,613
rank
55,523
PlumX Metrics