Asymptotically Optimal Adaptive A/B tests for Average Treatment Effect

88 Pages Posted: 22 Dec 2023

Date Written: November 24, 2023

Abstract

Typically in A/B testing, an experiment designer sequentially assigns treatment A or B to arriving individuals to identify the better treatment (known as the best treatment identification (BTI) problem), that is, the one with better mean performance. We focus on a related, equally important, but more informative problem of estimating the difference between the two means, defined as the average treatment effect (ATE). For computational efficiency, we restrict accuracy to a confidence interval (CI) of width at most ϵ, where the probability of CI not containing ATE is restricted to at most δ. The objective of the experiment designer is to estimate a CI of ATE with the minimum expected sample size, i.e., minimize the expected total number of individuals getting treatment A or B. We first establish a lower bound on the expected sample size of the A/B test (experiment) needed for any adaptive experimental policy, which constructs a CI of ATE with desired properties as the solution to a max-min problem for small δ. The min-max problem provides the asymptotic fraction of the assignment of treatments A and B for any asymptotically optimal policy. To gain further insights, we define a notion of information for any distribution. We prove that the fraction of treatment assignment for A and B should be inversely proportional to the square root of our proposed notion of information of the outcome distributions of treatments A and B, respectively. Using the insights provided by the max-min problem, we construct an adaptive policy that is asymptotically optimal, i.e., matches the lower bound on the expected sample size for small ϵ and δ. We compare our proposed policy with two popular policies: a) randomized controlled policies (where both treatments are given equal assignment) and b) assignment based on Neyman's rule (treatments are assigned proportional to their standard deviations) and show that there are meaningful gains from asymptotically optimal adaptive policies in terms of reducing the expected sample size of the A/B test. Finally, we present a comparative analysis between our ATE problem and the BTI problem revealing marked differences in the asymptotically optimal fraction of assignment of treatments in ATE and BTI problems.

Keywords: Sequential A/B Test, Valid Inference, Average Treatment Effect, Adaptive Experimental Design, Causal Inference, Best Treatment Identification, Confidence Interval

Suggested Citation

Deep, Vikas and Bassamboo, Achal and Juneja, Sandeep, Asymptotically Optimal Adaptive A/B tests for Average Treatment Effect (November 24, 2023). Available at SSRN: https://ssrn.com/abstract=4642974

Vikas Deep (Contact Author)

Northwestern University, Kellogg School of Management, Students ( email )

Evanston, IL
United States

Achal Bassamboo

Northwestern University - Kellogg School of Management ( email )

2001 Sheridan Road
Evanston, IL 60208
United States

Sandeep Juneja

Tata Institute of Fundamental Research (TIFR) ( email )

STCS, TIFR, 1, Homi Bhabha Road
Colaba
Mumbai, 400005
India

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
163
Abstract Views
1,058
Rank
380,072
PlumX Metrics