False Dichotomy Alert: Improving Subjective-Probability Estimates vs. Raising Awareness of Systemic Risk

Abstract

Taleb et al. (2022) portray the superforecasting research program as a masquerade that purports to build “survival functions for tail assessments via sports-like tournaments.’’ But that never was the goal. The program was designed to help intelligence analysts make better probability judgments, which required posing rapidly resolvable questions. From a signal detection theory perspective, the superforecasting and Taleb et al. programs are complementary, not contradictory (a point Taleb and Tetlock (2013) recognized). The superforecasting program aims at achieving high Hit rates at low cost in False-Positives whereas Taleb et al prioritize alerting us to systemic risk, even if that entails a high False-Positive rate. Proponents of each program should however acknowledge weaknesses in their cases. It is unclear: (a) how Taleb et al. (2022) can justify extreme error- avoidance trade-offs, without tacit probability judgments of rare, high-impact events; (b) how much superforecasting interventions can improve probability judgments of such events.

Keywords: superforecasting, systemic risk, fat-tailed dstributions, signal detection, forecasting tournaments, proper scoring rules