Utilizing External Predictions for Data Collection: Joint Optimization of Sampling and Measurement

Abstract

Recent advances in machine learning—especially large language models (LLMs)—enable low-cost, large-scale prediction of individual customer behaviors. While such predictions can reduce expensive costs in data collection, they may be systematically biased, resulting in undesired decisions. In this paper, we study how to optimally collect data with such predictions to preserve consistency and efficiency for estimating two target parameters—population mean and the average treatment effect. We propose a two-stage design that first samples units from the population according to a chosen density, and then for each sampled unit, either pays a fixed cost to observe the true outcome or relies on a free prediction.

For the non-adaptive setting with known joint distribution of predictions and outcomes, we derive the optimal sampling density and measurement probability by minimizing the semiparametric efficiency bound subject to a budget constraint. The solution follows a simple principle: sample and measure more where prediction residual variance is high or predictions are inaccurate. We then construct an efficient estimator that attains this bound. We also extend the framework to an adaptive setting in which predictors are iteratively updated from incoming data and distributional features are unknown ex ante; under standard regularity conditions, the design and estimator remain efficient and achieve the same bound. Empirical validations on two public-opinion survey datasets with LLM-generated predictions demonstrate that our approach attains comparable precision with roughly 40–50\% fewer true labels, or, at fixed budgets, reduces mean-squared error by about 40\% and tightens confidence intervals by around 30\%.

Keywords: Active learning, Adaptive experiments, Large language models