Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales

43 Pages Posted: 17 Mar 2021 Last revised: 27 Mar 2021

See all articles by Christopher Karvetski

Christopher Karvetski

affiliation not provided to SSRN

Carolyn Meinel

affiliation not provided to SSRN

Daniel Maxwell

affiliation not provided to SSRN

Yunzi Lu

University of Pennsylvania

Barb Mellers

University of Pennsylvania, Psychology; University of Pennsylvania, Wharton School

Philip Tetlock

University of Pennsylvania

Date Written: February 4, 2021

Abstract

Geopolitical forecasting tournaments have stimulated the development of methods for improving probability judgments of real-world events. But these innovations have focused on easier-to quantify variables, like personnel selection, training, teaming, and crowd aggregation—and bypassed messier constructs, like qualitative properties of forecasters’ rationales. Here we adapt methods from natural language processing (NLP) and computational text analysis to identify distinctive reasoning strategies in the rationales of top forecasters, including: (a) cognitive styles, such as dialectical complexity, that gauge tolerance of clashing perspectives and efforts to blend them into coherent conclusions; (b) the use of comparison classes or base rates to inform forecasts; (c) metrics derived from the Linguistic Inquiry and Word Count (LIWC) program. Applying these tools to multiple forecasting tournaments and to forecasters of widely varying skill (from Mechanical Turkers to carefully culled “superforecasters”) revealed that: (a) top forecasters show higher dialectical complexity in their rationales, use more comparison classes, and offer more past-focused rationales; (b) experimental interventions, like training and teaming, that boost accuracy also influence NLP profiles of rationales, nudging them in a “superforecaster-like” direction.

Keywords: geopolitical forecasting, psycholinguistic vocabularies, integrative complexity, comparison class, natural language processing, LIWC

Suggested Citation

Karvetski, Christopher and Meinel, Carolyn and Maxwell, Daniel and Lu, Yunzi and Mellers, Barb and Tetlock, Philip, Forecasting the Accuracy of Forecasters from Properties of Forecasting Rationales (February 4, 2021). Available at SSRN: https://ssrn.com/abstract=3779404 or http://dx.doi.org/10.2139/ssrn.3779404

Christopher Karvetski (Contact Author)

affiliation not provided to SSRN

Carolyn Meinel

affiliation not provided to SSRN

Daniel Maxwell

affiliation not provided to SSRN

Yunzi Lu

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Barb Mellers

University of Pennsylvania, Psychology ( email )

3815 Walnut Street
Philadelphia, PA 19104-6196
United States

University of Pennsylvania, Wharton School ( email )

3641 Locust Walk
Philadelphia, PA 19104-6365
United States

Philip Tetlock

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
334
Abstract Views
1,479
rank
125,759
PlumX Metrics