How Well Do Structural Demand Models Work? Counterfactual Predictions in School Choice

Abstract

This paper investigates the prediction accuracy of discrete choice models of school demand, using a policy reform in Boston that altered where applicants can apply under school choice. We find that the discrete choice models do not consistently outperform a much simpler heuristic, but their inconsistent performance largely arises from prediction errors in applicant characteristics, which are auxiliary inputs. Once we condition on the correct inputs, the discrete choice models consistently outperform, and their accuracy does not significantly improve upon refitting using post-reform data, suggesting that the choice models capture stable components of the preference distribution across policy regimes.

Keywords: discrete choice models, counterfactual predictions, simulation, out-of-sample validation, school choice

JEL Classification: C53, C25