Can Propensity Score Analysis Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within-Study Comparison

Abstract

The ability of propensity score analysis (PSA) to match impact estimates derived from random assignment (RA) is examined using data from the evaluation of two interdistrict magnet schools. As in previous within study comparisons, the estimates provided by PSA and RA differ substantially when PSA is implemented using comparison groups that are not similar to the treatment group and without pretreatment measures of academic performance. Adding pretreatment measures of the performance to the PSA, however, substantially improves the match between PSA and RA estimates. Although the results should not be generalized too readily, they suggest that non-experimental estimators can, in some circumstances, provide valid estimates of the causal impact of school choice programs.

Keywords: nonexperimental, quasi-experimental, propensity score analysis, design