Models, Markets, and Prediction Performance

26 Pages Posted: 5 Mar 2021 Last revised: 25 Jan 2022

See all articles by Rajiv Sethi

Rajiv Sethi

Barnard College, Columbia University; Santa Fe Institute

Julie Seager

Columbia University, Barnard College - Department of Economics

Emily Cai

Columbia University - The Fu Foundation School of Engineering and Applied Science

Daniel M. Benjamin

Nova Southeastern University; USC Information Sciences Institute

Fred Morstatter

USC Information Sciences Institute

Olivia Bobrownicki

Columbia University - Barnard College

Date Written: January 25, 2022

Abstract

Any forecasting model can be represented by a virtual trader endowed with a budget, risk preferences, and beliefs inherited from the model. We propose and implement a profitability test for the evaluation of forecasting models based on this idea. The virtual trader enters a position and adjusts its portfolio over time in response to changes in the model forecast and prediction market prices, and its eventual profitability can be used as a measure of model accuracy. We implement this test using probabilistic forecasts for thirteen battleground states in the 2020 US presidential election, using daily data from two sources over seven months: forecasts from a statistical model published by The Economist and prices from the PredictIt exchange. This analysis is then repeated using weekly data for the 2016 election. In both cases the model makes a modest profit, but there are interesting differences in the pattern of trade. The proposed approach can be applied more generally to any forecasting activity, as long as models and markets referencing the same events exist. The approach can also be used for comparative model evaluation, and for the construction of hybrid prediction markets in which the model acts as a market maker, providing liquidity, narrowing spreads, and making human participation more attractive.

Keywords: Prediction Markets, Structural Models, Forecast Evaluation, Presidential Elections

JEL Classification: D83, D84, G13

Suggested Citation

Sethi, Rajiv and Seager, Julie and Cai, Emily and Benjamin, Daniel and Morstatter, Fred and Bobrownicki, Olivia, Models, Markets, and Prediction Performance (January 25, 2022). Available at SSRN: https://ssrn.com/abstract=3767544 or http://dx.doi.org/10.2139/ssrn.3767544

Rajiv Sethi (Contact Author)

Barnard College, Columbia University ( email )

3009 Broadway
New York, NY 10027
United States
212-854-5140 (Phone)

HOME PAGE: http://www.columbia.edu/~rs328/

Santa Fe Institute

1399 Hyde Park Road
Santa Fe, NM 87501
United States

Julie Seager

Columbia University, Barnard College - Department of Economics ( email )

3009 Broadway
New York, NY 10027
United States

Emily Cai

Columbia University - The Fu Foundation School of Engineering and Applied Science ( email )

New York, NY 10027
United States

Daniel Benjamin

Nova Southeastern University ( email )

Ft. Lauderdale, FL 33314
United States
9542625012 (Phone)

USC Information Sciences Institute ( email )

4676 Admiralty Way
Suite 1001
Marina del Rey, CA 90292
United States

Fred Morstatter

USC Information Sciences Institute ( email )

4676 Admiralty Way
Suite 1001
Marina del Rey, CA 90292
United States

Olivia Bobrownicki

Columbia University - Barnard College ( email )

3009 Broadway
New York, NY 10027
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
237
Abstract Views
1,324
Rank
198,978
PlumX Metrics