Does Peer Review Work? An Experiment of Experimentalism
82 Pages Posted: 29 May 2016
Date Written: May 27, 2016
Ensuring the accuracy and consistency of highly decentralized and discretionary decision making is a core challenge for the administrative state. The widely influential school of “democratic experimentalism” posits that peer review provides a way forward, but systematic evidence remains limited. This Article provides the first empirical study of the feasibility and effects of peer review as a governance mechanism, based on a unique randomized controlled trial conducted with the largest health department in Washington State (Public Health-Seattle and King County). We randomly assigned half of the food safety inspection staff to engage in an intensive peer review process for a 4-month period. Pairs of inspectors jointly visited establishments, separately assessed health code violations, and deliberated about divergences on health code implementation. Our findings are threefold. First, observing identical conditions, inspectors disagreed 60% of the time. These joint inspection results in turn helped to pinpoint challenging code items and to efficiently develop training and guidance documents during weekly sessions. Second, analyzing over 28,000 independently conducted inspections across the peer review and control groups, we find that the intervention caused an increase in violations detected and scored by 17-19%. Third, peer review appeared to decrease variability across inspectors, thereby improving the consistency of inspections. As a result of this trial, King County has now instituted peer review as a standard practice. Our study has rich implications for the feasibility, promise, practice, and pitfalls of peer review, democratic experimentalism, and the administrative state.
Keywords: administrative law, randomized controlled trial, peer review, food safety, inspections, democratic experimentalism
Suggested Citation: Suggested Citation