Using Lagged Outcomes to Evaluate Bias in Value-Added Models

16 Pages Posted: 8 Feb 2016 Last revised: 4 Jul 2024

See all articles by Raj Chetty

Raj Chetty

Harvard University

John Friedman

Brown University

Jonah E. Rockoff

Columbia University - Columbia Business School, Finance; National Bureau of Economic Research (NBER)

Date Written: February 2016

Abstract

Value-added (VA) models measure the productivity of agents such as teachers or doctors based on the outcomes they produce. The utility of VA models for performance evaluation depends on the extent to which VA estimates are biased by selection, for instance by differences in the abilities of students assigned to teachers. One widely used approach for evaluating bias in VA is to test for balance in lagged values of the outcome, based on the intuition that today’s inputs cannot influence yesterday’s outcomes. We use Monte Carlo simulations to show that, unlike in conventional treatment effect analyses, tests for balance using lagged outcomes do not provide robust information about the degree of bias in value-added models for two reasons. First, the treatment itself (value-added) is estimated, rather than exogenously observed. As a result, correlated shocks to outcomes can induce correlations between current VA estimates and lagged outcomes that are sensitive to model specification. Second, in most VA applications, estimation error does not vanish asymptotically because sample sizes per teacher (or principal, manager, etc.) remain small, making balance tests sensitive to the specification of the error structure even in large datasets. We conclude that bias in VA models is better evaluated using techniques that are less sensitive to model specification, such as randomized experiments, rather than using lagged outcomes.

Suggested Citation

Chetty, Raj and Friedman, John and Rockoff, Jonah E., Using Lagged Outcomes to Evaluate Bias in Value-Added Models (February 2016). NBER Working Paper No. w21961, Available at SSRN: https://ssrn.com/abstract=2729061

Raj Chetty (Contact Author)

Harvard University ( email )

1875 Cambridge Street
Cambridge, MA 02138
United States

John Friedman

Brown University ( email )

Box 1860
Providence, RI 02912
United States

Jonah E. Rockoff

Columbia University - Columbia Business School, Finance ( email )

3022 Broadway
New York, NY 10027
United States

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
22
Abstract Views
399
PlumX Metrics