Multi-armed Bandit Experimental Design: Online Decision-making and Adaptive Inference

52 Pages Posted: 28 Sep 2022 Last revised: 2 Nov 2023

See all articles by David Simchi-Levi

David Simchi-Levi

Massachusetts Institute of Technology (MIT) - School of Engineering

Chonghuan Wang

Massachusetts Institute of Technology (MIT)

Date Written: September 20, 2022

Abstract

Multi-armed bandit has been well-known for its efficiency in online decision-making in terms of minimizing the loss of the participants' welfare during experiments (i.e., the regret). In clinical trials and many other scenarios, the statistical power of inferring the treatment effects (i.e., the gaps between the mean outcomes of different arms) is also crucial. Nevertheless, minimizing the regret entails harming the statistical power of estimating the treatment effect, since the observations from some arms can be limited. In this paper, we investigate the trade-off between efficiency and statistical power by casting the multi-armed bandit experimental design into a minimax multi-objective optimization problem. We introduce the concept of Pareto optimality to mathematically characterize the situation in which neither the statistical power nor the efficiency can be improved without degrading the other. We derive a useful sufficient and necessary condition for the Pareto optimal solutions to the minimax multi-objective optimization problem. Additionally, we design an effective Pareto optimal multi-armed bandit experiment that can be tailored to different levels of the trade-off between the two objectives. Moreover, we extend the design and analysis to the setting where the outcome of each arm consists of an adversarial baseline reward and a stochastic treatment effect, demonstrating the robustness of our design. Finally, motivated by practical applications, we examine the setting where the employed experiment must spilt the experimental units into a small number of batches. The outcomes of each batch can only be observed and utilized after the whole batch being experimented.

Keywords: Adaptive Experimental Design, Multi-armed Bandit, Online Learning, Treatment Effect, Minimax Multi-Objective Optimization, Pareto Optimality

Suggested Citation

Simchi-Levi, David and Wang, Chonghuan, Multi-armed Bandit Experimental Design: Online Decision-making and Adaptive Inference (September 20, 2022). Available at SSRN: https://ssrn.com/abstract=4224969 or http://dx.doi.org/10.2139/ssrn.4224969

David Simchi-Levi

Massachusetts Institute of Technology (MIT) - School of Engineering ( email )

MA
United States

Chonghuan Wang (Contact Author)

Massachusetts Institute of Technology (MIT) ( email )

77 Massachusetts Avenue
50 Memorial Drive
Cambridge, MA 02139-4307
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
668
Abstract Views
1,765
Rank
73,155
PlumX Metrics