Getting the Most Out of A/B Tests Using the Asymptotic Minimax-Regret Criteria

54 Pages Posted: 13 Jan 2023 Last revised: 21 Jan 2025

See all articles by Joonhwi Joo

Joonhwi Joo

University of Texas at Dallas - Naveen Jindal School of Management

Khai Chiong

affiliation not provided to SSRN

Date Written: January 20, 2025

Abstract

Many firms conduct A/B tests to find a marketing action that improves a value of interest, such as revenue or profit. We develop the asymptotic minimax regret (AMMR) criterion, a practical decision-theoretic approach for choosing among binary marketing actions based on A/B tests. The AMMR is a general large-sample approximation of the minimax-regret criterion from a frequentist standpoint. Our method directly optimizes the decision-relevant metric, accounting for the product of the error probability and the associated magnitude of value loss. Implementing the AMMR decision rule is straightforward; it comprises simply comparing the standardized treatment-effect estimate to the AMMR-optimal decision threshold. The AMMR suggests selecting the treatment whenever the point estimate is positive, as this minimizes the maximum expected net loss from decision errors. We illustrate our approach with a case study of a mobile game company's A/B testing with Monte Carlo validation, demonstrating that the AMMR decision rule effectively selects the optimal marketing action and improves revenue across a wide range of data-generating processes.

Keywords: minimax regret, decision theory, A/B tests

Suggested Citation

Joo, Joonhwi and Chiong, Khai, Getting the Most Out of A/B Tests Using the Asymptotic Minimax-Regret Criteria (January 20, 2025). Available at SSRN: https://ssrn.com/abstract=4323382 or http://dx.doi.org/10.2139/ssrn.4323382

Joonhwi Joo (Contact Author)

University of Texas at Dallas - Naveen Jindal School of Management ( email )

P.O. Box 830688
Richardson, TX TX 75083-0688
United States
75080 (Fax)

Khai Chiong

affiliation not provided to SSRN

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
298
Abstract Views
950
Rank
210,036
PlumX Metrics