Distilling the Wisdom of Crowds: Prediction Markets versus Prediction Polls

Management Science, Early View, Forthcoming

37 Pages Posted: 15 Sep 2015 Last revised: 17 Jun 2016

Pavel D. Atanasov

University of Pennsylvania

Phillip Rescober

Independent

Eric Stone

Independent

Samuel A. Swift

University of California, Berkeley - Institute of Business and Economic Research (IBER); Tepper School of Business

Emile Servan-Schreiber

LUMENOGIC

Philip E. Tetlock

University of Pennsylvania

Lyle Ungar

University of Pennsylvania

Barbara Mellers

University of California, Berkeley - Marketing Group; University of Pennsylvania - Marketing Department

Date Written: October 8, 2015

Abstract

We report the results of the first large-scale, long-term, experimental test between two crowd sourcing methods – prediction markets and prediction polls. More than 2,400 participants made forecasts on 261 events over two seasons of a geopolitical prediction tournament. Some forecasters traded in a continuous double auction market and were ranked based on earnings. Others submitted probability judgments, independently or in teams, and were ranked based on Brier scores. In both seasons of the tournament, last day prices from the prediction market were more accurate than the simple mean of forecasts from prediction polls. However, team prediction polls outperformed prediction markets when poll forecasts were aggregated with algorithms using temporal decay, performance weighting and recalibration. The biggest advantage of prediction polls occurred at the start of long-duration questions. Prediction polls with proper scoring, algorithmic aggregation and teaming offer an attractive alternative to prediction markets for distilling the wisdom of crowds.

Keywords: Prediction, Forecasting, Prediction Markets, Crowdsourcing

JEL Classification: C82, C93, D84

Suggested Citation

Atanasov, Pavel D. and Rescober, Phillip and Stone, Eric and Swift, Samuel A. and Servan-Schreiber, Emile and Tetlock, Philip E. and Ungar, Lyle and Mellers, Barbara, Distilling the Wisdom of Crowds: Prediction Markets versus Prediction Polls (October 8, 2015). Management Science, Early View, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2660628

Pavel D. Atanasov (Contact Author)

University of Pennsylvania ( email )

3815 Walnut Street
Philadelphia, PA 19104-6196
United States

Phillip Rescober

Independent ( email )

No Address Available

Eric Stone

Independent ( email )

No Address Available

Samuel A. Swift

University of California, Berkeley - Institute of Business and Economic Research (IBER) ( email )

F502 Haas Building
Berkeley, CA 94720-1922
United States

Tepper School of Business ( email )

5000 Forbes Avenue
Pittsburgh, PA 15213-3890
United States

Emile Servan-Schreiber

LUMENOGIC ( email )

475 Longview Road
South Orange, NJ 07079
United States

Philip E. Tetlock

University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Lyle Ungar

University of Pennsylvania

Philadelphia, PA 19104
United States

Barbara Mellers

University of California, Berkeley - Marketing Group ( email )

Haas School of Business
Berkeley, CA 94720
United States

University of Pennsylvania - Marketing Department

700 Jon M. Huntsman Hall
3730 Walnut Street
Philadelphia, PA 19104-6340
United States

Paper statistics

Downloads
278
Rank
87,769
Abstract Views
1,399