Policy Choice and the Wisdom of Crowds

27 Pages Posted: 14 Sep 2022

See all articles by Nicholas Otis

Nicholas Otis

University of California Berkeley

Date Written: August 25, 2022


Using data from seven large-scale randomized experiments, I test whether crowds of academic experts can forecast the relative effectiveness of policy interventions. Eight-hundred and sixty-three academic experts provided 9,295 forecasts of the causal effects from these experiments, which span a diverse set of interventions (e.g., information provision, psychotherapy, soft-skills training), outcomes (e.g., consumption, COVID-19 vaccination, employment), and locations (Jordan, Kenya, Sweden, the United States). For each policy comparisons (a pair of policies and an outcome), I calculate the percent of crowd forecasts that correctly rank policies by their experimentally estimated treatment effects. While only 65% of individual experts identify which of two competing policies will have a larger causal effect, the average forecast from bootstrapped crowds of 30 experts identifies the better policy 86% of the time, or 92% when restricting analysis to pairs of policies who effects differ at the p < 0.10 level. Only 10 experts are needed to produce an 18-percentage point (27%) improvement in policy choice.

Keywords: wisdom of crowds, forecasting, prediction, judgement, crowdsourcing

JEL Classification: A14, C53, C90, O10

Suggested Citation

Otis, Nicholas, Policy Choice and the Wisdom of Crowds (August 25, 2022). Available at SSRN: https://ssrn.com/abstract=4200841 or http://dx.doi.org/10.2139/ssrn.4200841

Nicholas Otis (Contact Author)

University of California Berkeley ( email )

United States

HOME PAGE: http://www.nicholasotis.com

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
PlumX Metrics