Does Peer Review Work? An Experiment of Experimentalism

82 Pages Posted: 29 May 2016

Date Written: May 27, 2016

Abstract

Ensuring the accuracy and consistency of highly decentralized and discretionary decision making is a core challenge for the administrative state. The widely influential school of “democratic experimentalism” posits that peer review provides a way forward, but systematic evidence remains limited. This Article provides the first empirical study of the feasibility and effects of peer review as a governance mechanism, based on a unique randomized controlled trial conducted with the largest health department in Washington State (Public Health-Seattle and King County). We randomly assigned half of the food safety inspection staff to engage in an intensive peer review process for a 4-month period. Pairs of inspectors jointly visited establishments, separately assessed health code violations, and deliberated about divergences on health code implementation. Our findings are threefold. First, observing identical conditions, inspectors disagreed 60% of the time. These joint inspection results in turn helped to pinpoint challenging code items and to efficiently develop training and guidance documents during weekly sessions. Second, analyzing over 28,000 independently conducted inspections across the peer review and control groups, we find that the intervention caused an increase in violations detected and scored by 17-19%. Third, peer review appeared to decrease variability across inspectors, thereby improving the consistency of inspections. As a result of this trial, King County has now instituted peer review as a standard practice. Our study has rich implications for the feasibility, promise, practice, and pitfalls of peer review, democratic experimentalism, and the administrative state.

Keywords: administrative law, randomized controlled trial, peer review, food safety, inspections, democratic experimentalism

Suggested Citation

Ho, Daniel E., Does Peer Review Work? An Experiment of Experimentalism (May 27, 2016). Stanford Law Review, Vol. 69, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2785927

Daniel E. Ho (Contact Author)

Stanford Law School ( email )

559 Nathan Abbott Way
Stanford, CA 94305-8610
United States
650-723-9560 (Phone)

HOME PAGE: http://dho.stanford.edu

Register to save articles to
your library

Register

Paper statistics

Downloads
271
Abstract Views
1,667
rank
114,379
PlumX Metrics