Estimating Effects of Long-Term Treatments

Abstract

One lingering challenge of randomized controlled trials (or A/B tests) is to estimate the effects of long-term treatments at an early stage of the experiment. Learning such effects is crucial for management, as product updates (e.g., new UIs or algorithms) are intended to remain in the system for a long time once implemented but conducting long-duration experiments is costly. In this paper, we propose a longitudinal surrogate model to estimate the effects of long-term treatments using data collected from short-term experiments and historical observations. We show that under standard assumptions, the effect of long-term treatments can be decomposed into a sequence of functions that depend on the user attributes, their short-term intermediate metrics, and the treatment assignments. We describe three sets of identification assumptions that each leads to one estimation strategy, and discuss the advantages and limitations of each estimation strategy. We conduct two large-scale long-term experiments on WeChat, an instant messaging platform, and demonstrate the effectiveness of our methods. For practitioners, using our methods could significantly reduce the amount of time required to conduct experiments.

Keywords: Long-term effect, statistical surrogate, A/B testing