Automated Experimental Design with Optimization from Historical Data Simulations

Abstract

We study the problem of designing randomized experiments to estimate the causal effects of new interventions. The design problem involves selecting a treatment assignment mechanism, including assignment probabilities, in order to optimize a user-defined objective (e.g., maximizing the precision of a causal-effect estimator). A key challenge is that the objective is typically a black-box function of the design, depending on unknown outcome-generating processes and causal effects. To tackle this challenge, we propose a novel automated experimental design (Auto-EXD) approach. Auto-EXD evaluates the objective value of a candidate design by simulating experiments based on stationary historical control data and prior distributions of causal effects. Then a gradient-free method is used to iteratively optimize the design. We rigorously analyze the convergence behavior of Auto-EXD and show how it depends on temporal and cross-unit correlations in historical data as well as on implementation specifics. In synthetic experiments on three application domains--digital platforms, health, and energy--Auto-EXD can reduce the estimation error of treatment effects by up to 25% compared to the state-of-the-art benchmark designs. Our results reveal new insights into improving design efficiency. In particular, for multi-unit crossover experiments with multiple interventions, we find that efficiency is improved by sequentially rolling out (a) the same intervention across units and (b) different interventions on the same unit.

Keywords: multi-period experiment, causal effect, black-box model, gradient-free optimization, digital platform