Non-stationary A/B Tests: Optimal Variance Reduction, Bias Correction, and Valid Inference

Abstract

We develop an analytical framework to appropriately model and adequately analyze A/B tests in presence of nonparametric non-stationarities in the targeted business metrics. A/B tests, also known as online randomized controlled experiments, have been used at scale by data-driven enterprises to guide decisions and test innovative ideas to improve core business metrics. Meanwhile, non-stationarities, such as the time-of-day effect and the day-of-week effect, can often arise nonparametrically in key business metrics involving purchases, revenue, conversions, customer experiences, etc. First, we develop a generic nonparametric stochastic model to capture nonstationarities in A/B test experiments. We build a practically relevant limiting regime to facilitate analyzing large-sample estimator performances under nonparametric non-stationarities. Second, we show that ignoring or inadequately addressing non-stationarities can cause standard A/B tests estimators to have sub-optimal variance and non-vanishing bias, therefore leading to loss of statistical efficiency and accuracy. We provide a new estimator that views time as a continuous strata and performs post stratification with a data-dependent number of stratification levels. Without making parametric assumptions, we prove a central limit theorem for the proposed estimator and show that the estimator attains the best achievable asymptotic variance and is asymptotically unbiased. Third, we propose a time-grouped randomization that is designed to balance treatment and control assignments at granular time scales. We show that when the time-grouped randomization is integrated to standard experimental designs to generate experiment data, simple A/B test estimators can achieve asymptotically optimal variance. A brief account of numerical experiments are conducted to illustrate the analysis.

Keywords: A/B tests, non-stationarity, central limit theorem, optimal asymptotic variance, bias, inference