Policy-Aware Experimentation: Strategic Sampling for Optimized Targeting Policies

Abstract

With unprecedented access to consumer information, firms are increasingly interested in designing highly effective data-driven targeting policies based on detailed consumer data. The current standard for implementing such policies involves the "test-then-learn" approach, where randomized experiments are used to estimate the differential impact of marketing interventions on various customers. However, this method fails to incorporate the firm's ultimate business objectives, leading to inefficient experimentation and suboptimal targeting strategies. To overcome this limitation, we propose a sequential experimental design integrated with a novel sampling criterion—expected profit loss—which aligns theoretically with the firm's profit-maximizing objective. Additionally, we introduce a novel expected profit loss estimation method leveraging the power of Bayesian inference for uncertainty quantification based on Causal Forest. Through extensive simulation studies and two empirical applications, we demonstrate the superiority of our approach in improving targeting performance. Furthermore, we emphasize the effectiveness of our approach even when simplified into a two-stage design, enabling firms to shorten the experimentation period and streamline the process. Our research underscores the importance of aligning experimental design with business objectives and offers an efficient solution for firms seeking to enhance their targeting strategies.

Keywords: Policy learning, marketing interventions, targeted policies, experimentation, active learning, heterogeneous treatment effect