Calibration of Heterogeneous Treatment Effects in Random Experiments

58 Pages Posted: 8 Jul 2021 Last revised: 27 Jun 2023

See all articles by Yan Leng

Yan Leng

University of Texas at Austin - Red McCombs School of Business

Drew Dimmery

affiliation not provided to SSRN

Date Written: June 28, 2021

Abstract

Machine learning is commonly used to estimate the heterogeneous treatment effects (HTEs) in randomized experiments. Using a large-scale randomized experiment on Facebook, we observe a substantial discrepancy between machine learning-based treatment effect estimates and difference-in-means estimates directly from the randomized experiment. This paper provides a two-step framework for practitioners and researchers to diagnose and correct this discrepancy. We propose a diagnostic tool to assess whether bias exists in the model-based estimates from machine learning. If bias exists, we provide a model-agnostic method to calibrate any HTE estimates to known, unbiased, subgroup difference-in-means estimates, ensuring that the sign and magnitude of the estimates approximate the model-free benchmarks. Our method requires no additional data beyond the data necessary for estimating HTEs, and it can be scaled to arbitrarily large datasets. We use synthetic simulations to explain two sources of bias: misspecification bias and regularization bias. We illustrate the effectiveness of our calibration method using extensive synthetic simulations and two real-world randomized experiments. We further demonstrate the practical value of this calibration in three general and canonical policy-making settings: in a prescriptive, budget-constrained optimization framework; in a setting seeking to maximize multiple performance indicators; and in a multi-treatment uplift modeling setting.

Keywords: Causal inference, Heterogeneous treatment effects, Random experiments, Calibration, Machine learning

Suggested Citation

Leng, Yan and Dimmery, Drew, Calibration of Heterogeneous Treatment Effects in Random Experiments (June 28, 2021). Available at SSRN: https://ssrn.com/abstract=3875850 or http://dx.doi.org/10.2139/ssrn.3875850

Yan Leng (Contact Author)

University of Texas at Austin - Red McCombs School of Business ( email )

Austin, TX
United States

Drew Dimmery

affiliation not provided to SSRN

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
340
Abstract Views
1,089
Rank
165,430
PlumX Metrics