Quantifying Robustness to External Validity Bias

Abstract

The external validity of experimental results is essential in the social sciences. Existing methods estimate causal effects in a target population, called the target population average treatment effect (T-PATE). However, these methods are sometimes difficult to implement either because it is infeasible to obtain data for the target population or because there is no target population that analysts and skeptics can agree on. We consider a different goal — quantifying how robust an experiment is to external validity bias. In particular, we propose a measure of external robustness by estimating how much different a population should be from the experimental sample to explain away the T-PATE. Large estimated external robustness implies that causal conclusions remain the same unless populations of interest are significantly different from the experimental sample. Unlike the standard generalization approach, estimation of external robustness only requires experimental data and does not require any population data. We prove that the proposed estimator is consistent to the true external robustness under common generalization assumptions and, more importantly, has a simple interpretation even when those assumptions are violated. We provide benchmarks to help interpret the degree of external robustness in each application.

Keywords: Causal inference, External validity, Experiment, Generalization

JEL Classification: C9