Adaptive Neyman Allocation

Abstract

In experimental design, Neyman allocation refers to the practice of allocating subjects into treated and control groups, in possibly unequal numbers that are proportional to their respective standard deviations, with the objective of minimizing the variance of the treatment effect estimator. This widely recognized approach increases statistical power in scenarios where the sample size is limited, as is often the case in social experiments, clinical trials, and marketing research. However, Neyman allocation cannot be implemented unless the standard deviations are known in advance. Fortunately, the multi-stage nature of the aforementioned applications allows the use of earlier stage observations to estimate the standard deviations, which further guide the allocation in later stages. In this paper, we introduce a competitive analysis framework to study this multi-stage experimental design problem and proposes a simple, near-optimal adaptive Neyman allocation algorithm. Our result nearly matches the information-theoretic limit of conducting experiments, making our algorithm a simple and effective solution for multi-stage experimental designs.