Asymptotically Optimal Adaptive A/B tests for Average Treatment Effect

Abstract

Typically in A/B testing, an experiment designer sequentially assigns treatment A or B to arriving individuals to identify the better treatment (known as the best treatment identification (BTI) problem), that is, the one with better mean performance. We focus on a related, equally important, but more informative problem of estimating the difference between the two means, defined as the average treatment effect (ATE). For computational efficiency, we restrict accuracy to a confidence interval (CI) of width at most ϵ, where the probability of CI not containing ATE is restricted to at most δ. The objective of the experiment designer is to estimate a CI of ATE with the minimum expected sample size, i.e., minimize the expected total number of individuals getting treatment A or B. We first establish a lower bound on the expected sample size of the A/B test (experiment) needed for any adaptive experimental policy, which constructs a CI of ATE with desired properties as the solution to a max-min problem for small δ. The min-max problem provides the asymptotic fraction of the assignment of treatments A and B for any asymptotically optimal policy. To gain further insights, we define a notion of information for any distribution. We prove that the fraction of treatment assignment for A and B should be inversely proportional to the square root of our proposed notion of information of the outcome distributions of treatments A and B, respectively. Using the insights provided by the max-min problem, we construct an adaptive policy that is asymptotically optimal, i.e., matches the lower bound on the expected sample size for small ϵ and δ. We compare our proposed policy with two popular policies: a) randomized controlled policies (where both treatments are given equal assignment) and b) assignment based on Neyman's rule (treatments are assigned proportional to their standard deviations) and show that there are meaningful gains from asymptotically optimal adaptive policies in terms of reducing the expected sample size of the A/B test. Finally, we present a comparative analysis between our ATE problem and the BTI problem revealing marked differences in the asymptotically optimal fraction of assignment of treatments in ATE and BTI problems.

Keywords: Sequential A/B Test, Valid Inference, Average Treatment Effect, Adaptive Experimental Design, Causal Inference, Best Treatment Identification, Confidence Interval