Test & Roll: Profit-Maximizing A/B Tests

46 Pages Posted: 1 Nov 2018 Last revised: 23 May 2019

See all articles by Elea McDonnell Feit

Elea McDonnell Feit

Drexel University - Department of Marketing

Ron Berman

University of Pennsylvania - The Wharton School

Date Written: May 21, 2019


Marketers often use A/B testing as a tool to compare marketing treatments in a test stage and then deploy the better-performing treatment to the remainder of the consumer population. While these tests have traditionally been analyzed using hypothesis testing, we re-frame them as an explicit trade-off between the opportunity cost of the test (where some customers receive a sub-optimal treatment) and the potential losses associated with deploying a sub-optimal treatment to the remainder of the population.

We derive a closed-form expression for the profit-maximizing test size and show that it is substantially smaller than typically recommended for a hypothesis test, particularly when the response is noisy or when the total population is small. The common practice of using small holdout groups can be rationalized by asymmetric priors. The proposed test design achieves nearly the same expected regret as the flexible, yet harder-to-implement multi-armed bandit under a wide range of conditions.

We demonstrate the benefits of the method in three different marketing contexts -- website design, display advertising and catalog tests -- in which we estimate priors from past data. In all three cases, the optimal sample sizes are substantially smaller than for a traditional hypothesis test, resulting in higher profit.

Keywords: A/B Testing, Randomized Controlled Trial, Marketing Experiments, Bayesian Decision Theory, Sample Size

JEL Classification: C11, C12, C44, C61, C83, C9, M31

Suggested Citation

Feit, Elea McDonnell and Berman, Ron, Test & Roll: Profit-Maximizing A/B Tests (May 21, 2019). Available at SSRN: https://ssrn.com/abstract=3274875 or http://dx.doi.org/10.2139/ssrn.3274875

Elea McDonnell Feit (Contact Author)

Drexel University - Department of Marketing ( email )

United States

Ron Berman

University of Pennsylvania - The Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Abstract Views
PlumX Metrics