The Central Role of Noise in Evaluating Interventions that Use Test Scores to Rank Schools

32 Pages Posted: 27 Nov 2003 Last revised: 26 Aug 2022

See all articles by Kenneth Y. Chay

Kenneth Y. Chay

University of California, Berkeley - Department of Economics; National Bureau of Economic Research (NBER)

Patrick J. McEwan

Wellesley College, Department of Economics

Miguel S. Urquiola

Columbia University

Date Written: November 2003

Abstract

Several countries have implemented programs that use test scores to rank schools, and to reward or penalize them based on their students' average performance. Recently, Kane and Staiger (2002) have warned that imprecision in the measurement of school-level test scores could impede these efforts. There is little evidence, however, on how seriously noise hinders the evaluation of the impact of these interventions. We examine these issues in the context of Chile's P-900 program a country-wide intervention in which resources were allocated based on cutoffs in schools' mean test scores. We show that transitory noise in average scores and mean reversion lead conventional estimation approaches to greatly overstate the impacts of such programs. We then show how a regression discontinuity design that utilizes the discrete nature of the selection rule can be used to control for reversion biases. While the RD analysis provides convincing evidence that the P-900 program had significant effects on test score gains, these effects are much smaller than is widely believed.

Suggested Citation

Chay, Kenneth Y. and McEwan, Patrick J. and Urquiola, Miguel S., The Central Role of Noise in Evaluating Interventions that Use Test Scores to Rank Schools (November 2003). NBER Working Paper No. w10118, Available at SSRN: https://ssrn.com/abstract=471468

Kenneth Y. Chay (Contact Author)

University of California, Berkeley - Department of Economics ( email )

549 Evans Hall #3880
Berkeley, CA 94720-3880
United States
510-643-8596 (Phone)
510-642-0638 (Fax)

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Patrick J. McEwan

Wellesley College, Department of Economics ( email )

106 Central Street
Wellesley, MA 02481
United States

HOME PAGE: http://www.patrickmcewan.net

Miguel S. Urquiola

Columbia University ( email )

420 W. 118th Street
New York, NY 10027
United States