Reciprocal Scoring: A Method for Forecasting Unanswerable Questions

68 Pages Posted: 18 Nov 2021

See all articles by Ezra Karger

Ezra Karger

Federal Reserve Bank of Chicago

Joshua Monrad

University of Oxford - Future of Humanity Institute

Barbara Mellers

University of Pennsylvania, Psychology; University of Pennsylvania, Wharton School

Philip Tetlock

University of Pennsylvania

Date Written: October 31, 2021

Abstract

We propose an elicitation method, Reciprocal Scoring (RS), that challenges forecasters to predict the forecasts of other forecasters. Two studies show how RS can generate accurate forecasts of otherwise unanswerable questions. Study 1 establishes the epistemic credibility of RS: forecasters randomly assigned to use RS were as accurate as forecasters predicting objectively resolvable outcomes using a proper scoring rule—and both groups were more accurate than a control group that felt accountable to neither intersubjective RS metrics nor objective metrics. Study 2 establishes the practical value of RS. We ask highly accurate forecasters to predict each other’s forecasts of the effect of government policies on COVID-19 mortality, yielding a real-time ranking of the expected effectiveness of pandemic-containment policies. As in Study 1, RS forecasters converged but in this case on policy recommendations that stand up to scrutiny, even with the benefit of hindsight. The core contribution of RS is its power to create accountability for accuracy in policy debates that have long been stalemated by the absence of accountability.

Keywords: forecasting tournaments, causal inference, policy evaluation, elicitation, COVID-19

Suggested Citation

Karger, Ezra and Monrad, Joshua and Mellers, Barbara and Tetlock, Philip, Reciprocal Scoring: A Method for Forecasting Unanswerable Questions (October 31, 2021). Available at SSRN: https://ssrn.com/abstract=3954498 or http://dx.doi.org/10.2139/ssrn.3954498

Ezra Karger (Contact Author)

Federal Reserve Bank of Chicago ( email )

230 South LaSalle Street
Chicago, IL 60604
United States

Joshua Monrad

University of Oxford - Future of Humanity Institute ( email )

Suite 8, Littlegate House
16/17 St Ebbe's Street
Oxford, OX1 1PT
United Kingdom

Barbara Mellers

University of Pennsylvania, Psychology ( email )

3815 Walnut Street
Philadelphia, PA 19104-6196
United States

University of Pennsylvania, Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Philip Tetlock

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
1,511
Abstract Views
5,743
Rank
23,076
PlumX Metrics