Evaluating Prediction Mechanisms: A Profitability Test

22 Pages Posted: 5 Mar 2021 Last revised: 30 May 2024

See all articles by Rajiv Sethi

Rajiv Sethi

Barnard College, Columbia University; Santa Fe Institute

Julie Seager

Massachusetts Institute of Technology (MIT) - Sloan School of Management

Emily Cai

CMU Tepper School of Business

Daniel M. Benjamin

Nova Southeastern University; USC Information Sciences Institute

Fred Morstatter

USC Information Sciences Institute

Olivia Bobrownicki

Columbia University - Barnard College

Yuqi Cheng

Columbia University, Barnard College - Department of Economics

Anushka Kumar

Columbia University - Barnard College

Anusha Wanganoo

Columbia University, Barnard College - Department of Economics

Date Written: January 25, 2022

Abstract

Any forecasting model can be represented by a virtual trader
in a prediction market, endowed with a budget, risk preferences,
and beliefs inherited from the model. We propose and
implement a profitability test for the evaluation of forecasting
models based on this idea. The virtual trader enters a position
and adjusts its portfolio over time in response to changes in
the model forecast and market prices, and its profitability can
be used as a measure of model accuracy. We implement this
test using probabilistic forecasts for competitive states in the
2020 US presidential election and congressional elections in
2020 and 2022, using data from three sources: model-based
forecasts published by The Economist and FiveThirtyEight, and
prices from the PredictIt exchange. The proposed approach
can be applied more generally to any forecasting activity as
long as models and markets referencing the same events exist

Keywords: Forecasting, Prediction Markets, Performance Evaluation

JEL Classification: D83, D84, G13

Suggested Citation

Sethi, Rajiv and Seager, Julie and Cai, Emily and Benjamin, Daniel and Morstatter, Fred and Bobrownicki, Olivia and Cheng, Yuqi and Kumar, Anushka and Wanganoo, Anusha, Evaluating Prediction Mechanisms: A Profitability Test (January 25, 2022). Available at SSRN: https://ssrn.com/abstract=3767544 or http://dx.doi.org/10.2139/ssrn.3767544

Rajiv Sethi (Contact Author)

Barnard College, Columbia University ( email )

3009 Broadway
New York, NY 10027
United States
212-854-5140 (Phone)

HOME PAGE: http://www.columbia.edu/~rs328/

Santa Fe Institute

1399 Hyde Park Road
Santa Fe, NM 87501
United States

Julie Seager

Massachusetts Institute of Technology (MIT) - Sloan School of Management ( email )

9175664890 (Phone)

Emily Cai

CMU Tepper School of Business ( email )

5000 Forbes Avenue
Pittsburgh, PA 15213
United States

Daniel Benjamin

Nova Southeastern University ( email )

Ft. Lauderdale, FL 33314
United States
9542625012 (Phone)

USC Information Sciences Institute ( email )

4676 Admiralty Way
Suite 1001
Marina del Rey, CA 90292
United States

Fred Morstatter

USC Information Sciences Institute ( email )

4676 Admiralty Way
Suite 1001
Marina del Rey, CA 90292
United States

Olivia Bobrownicki

Columbia University - Barnard College ( email )

3009 Broadway
New York, NY 10027
United States

Yuqi Cheng

Columbia University, Barnard College - Department of Economics ( email )

5281 Atlschul, 3009 Broadway
New York, NY 10027
United States

Anushka Kumar

Columbia University - Barnard College ( email )

3009 Broadway
New York, NY 10027
United States

Anusha Wanganoo

Columbia University, Barnard College - Department of Economics ( email )

3009 Broadway
New York, NY 10027
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
392
Abstract Views
1,905
Rank
142,597
PlumX Metrics