Models, Markets, and Prediction Performance

Abstract

Any forecasting model can be represented by a virtual trader endowed with a budget, risk preferences, and beliefs inherited from the model. We propose and implement a profitability test for the evaluation of forecasting models based on this idea. The virtual trader enters a position and adjusts its portfolio over time in response to changes in the model forecast and prediction market prices, and its eventual profitability can be used as a measure of model accuracy. We implement this test using probabilistic forecasts for thirteen battleground states in the 2020 US presidential election, using daily data from two sources over seven months: forecasts from a statistical model published by The Economist and prices from the PredictIt exchange. This analysis is then repeated using weekly data for the 2016 election. In both cases the model makes a modest profit, but there are interesting differences in the pattern of trade. The proposed approach can be applied more generally to any forecasting activity, as long as models and markets referencing the same events exist. The approach can also be used for comparative model evaluation, and for the construction of hybrid prediction markets in which the model acts as a market maker, providing liquidity, narrowing spreads, and making human participation more attractive.

Keywords: Prediction Markets, Structural Models, Forecast Evaluation, Presidential Elections

JEL Classification: D83, D84, G13