Enhancing External Validity in Experiments with Ongoing Sampling

Abstract

Participants in experiments often enroll over time, potentially compromising external validity due to temporal shifts in sample covariates. This issue is particularly significant in short-duration online experiments, such as A/B tests commonly used in product management. To address this, we introduce a novel framework that segments the ongoing sampling process into three distinct stages, each exhibiting varying degrees of sample representativeness and result generalizability for the population. The framework then creates stage-specific estimators for the Population Average Treatment Effect (PATE). By employing survival analysis, we develop a heuristic function to identify these stages without requiring prior knowledge of the population or sample characteristics. We validate our method using synthetic data and 600 real-world experiments conducted on WeChat, a leading social media platform. Our approach offers a cost-effective method to enhance the interpretability of sample representativeness and the generalizability of results in experiments with ongoing sampling. This enables managers to make more informed product decisions for the user population at any stage of the experiment.

Keywords: A/B testing, External Validity, Ongoing Sampling, Survival Analysis