Getting the Most Out of A/B Tests Using the Asymptotic Minimax-Regret Criteria
54 Pages Posted: 13 Jan 2023 Last revised: 21 Jan 2025
Date Written: January 20, 2025
Abstract
Many firms conduct A/B tests to find a marketing action that improves a value of interest, such as revenue or profit. We develop the asymptotic minimax regret (AMMR) criterion, a practical decision-theoretic approach for choosing among binary marketing actions based on A/B tests. The AMMR is a general large-sample approximation of the minimax-regret criterion from a frequentist standpoint. Our method directly optimizes the decision-relevant metric, accounting for the product of the error probability and the associated magnitude of value loss. Implementing the AMMR decision rule is straightforward; it comprises simply comparing the standardized treatment-effect estimate to the AMMR-optimal decision threshold. The AMMR suggests selecting the treatment whenever the point estimate is positive, as this minimizes the maximum expected net loss from decision errors. We illustrate our approach with a case study of a mobile game company's A/B testing with Monte Carlo validation, demonstrating that the AMMR decision rule effectively selects the optimal marketing action and improves revenue across a wide range of data-generating processes.
Keywords: minimax regret, decision theory, A/B tests
Suggested Citation: Suggested Citation