The A/B Test Deception: Divergent Delivery, Ad Response Heterogeneity, and Erroneous Inferences in Online Advertising Field Experiments

44 Pages Posted: 30 Jul 2021

See all articles by Michael Braun

Michael Braun

Southern Methodist University (SMU) Cox School of Business

Eric M. Schwartz

University of Michigan, Stephen M. Ross School of Business

Date Written: July 28, 2021

Abstract

Advertisers and researchers use tools provided by advertising platforms to conduct randomized experiments for testing user responses to creative elements in online ads. Internally valid comparisons between ads require the mix of experimental users exposed to each ad to be similar across all ads. But that internal validity is threatened when platforms' targeting algorithms deliver each ad to its own optimized mix of users, which diverges across ads. We extend the potential outcomes model of causal inference to treat random assignment of ads and the user exposure states for each ad as two separate decisions. We then demonstrate how targeting ads to users leads advertisers to incorrectly infer which ad performs better, based on aggregate test results. Through analysis and simulation, we characterize how bias in the aggregate estimate of the difference between two ads' lifts is driven by the interplay between heterogeneous responses to different ads and how platforms deliver ads to divergent subsets of users. We also identify conditions for an undetectable "Simpson's reversal," in which all unobserved types of users may prefer ad A over ad B, but the advertiser mistakenly infers from aggregate experimental results that users prefer ad B over ad A.

Keywords: Targeted online advertising, A/B testing, measuring advertising effectiveness, causal inference, experimental design, Simpson's paradox, social media

JEL Classification: C9, M31, M37

Suggested Citation

Braun, Michael and Schwartz, Eric M., The A/B Test Deception: Divergent Delivery, Ad Response Heterogeneity, and Erroneous Inferences in Online Advertising Field Experiments (July 28, 2021). SMU Cox School of Business Research Paper No. 21-10, Available at SSRN: https://ssrn.com/abstract=3896024 or http://dx.doi.org/10.2139/ssrn.3896024

Michael Braun (Contact Author)

Southern Methodist University (SMU) Cox School of Business ( email )

United States

Eric M. Schwartz

University of Michigan, Stephen M. Ross School of Business ( email )

701 Tappan Street
Ann Arbor, MI MI 48109
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
66
Abstract Views
477
rank
419,947
PlumX Metrics