External Validity Bias of Purposive and Random Site Selection When Sites Can Opt Out: Evidence from the Head Start Program

Abstract

While randomized controlled trials (RCTs) produce impact evidence with high internal validity, they are often conducted in purposive samples that may poorly represent the population of interest. Random site selection has been proposed as an alternative for reducing "external validity bias" in estimating the Population Average Treatment Effect. However, there is little prior research on the performance of random and other site selection methods when selected sites may decline to participate. This study conducted simulations to estimate external validity bias in an RCT of the Head Start program. Purposive site selection yielded external validity bias that varied depending on the outcome examined, the factors used to purposively select sites, and the assumptions used to simulate site decisions about whether to participate. Simple random site selection yielded less external validity bias than purposive site selection under some but not all conditions. Stratified random site selection yielded the least external validity bias, but the results likely overstate the performance when data on impact moderators are not available initially before sites are selected. The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305D190020 to Westat. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education. The authors would like to thank Elizabeth Petraglia for her contributions to an early draft of the paper. We would also like to thank ICPSR for providing access to the data used in the analysis.