Where A-B Testing Goes Wrong: What Online Experiments Cannot (and Can) Tell You About How Customers Respond to Advertising

59 Pages Posted: 30 Jul 2021 Last revised: 25 May 2023

See all articles by Michael Braun

Michael Braun

Southern Methodist University (SMU) - Marketing Department

Eric M. Schwartz

University of Michigan, Stephen M. Ross School of Business

Date Written: May 25, 2023

Abstract

Marketers use online advertising platforms to compare user responses to different ad content. However, platforms’ experimentation tools deliver ads to distinct, optimized, undetectable mixes of users that vary across ads, even during the test. As a result, the estimated A-B comparison from the data reflects the combination of ad content and algorithmic selection of users, which is different than what would have occurred under random exposure. We empirically demonstrate this “divergent delivery” pattern using data from an A-B test that we ran on a major ad platform. This paper explains how algorithmic targeting, user heterogeneity, and data aggregation conspire to confound the magnitude, and even the sign, of ad A-B test results, and what the implications are for different roles in the marketing organization with varying experimentation goals. We also consider the counterfactual case of disabling divergent delivery, where user types are balanced across ads. By extending the potential outcomes model of causal inference, we treat random assignment of ads and user exposure to ads as independent decisions. Since not all marketers have the same decision-making goals for these ad A-B tests, we offer prescriptive guidance to experimenters based on their needs.

Keywords: Targeted online advertising, A/B testing, measuring advertising effectiveness, causal inference, experimental design, Simpson's paradox, social media

JEL Classification: C9, M31, M37

Suggested Citation

Braun, Michael and Schwartz, Eric M., Where A-B Testing Goes Wrong: What Online Experiments Cannot (and Can) Tell You About How Customers Respond to Advertising (May 25, 2023). SMU Cox School of Business Research Paper No. 21-10, Available at SSRN: https://ssrn.com/abstract=3896024 or http://dx.doi.org/10.2139/ssrn.3896024

Michael Braun (Contact Author)

Southern Methodist University (SMU) - Marketing Department ( email )

United States

Eric M. Schwartz

University of Michigan, Stephen M. Ross School of Business ( email )

701 Tappan Street
Ann Arbor, MI MI 48109
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
519
Abstract Views
2,163
Rank
94,450
PlumX Metrics