Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents

Abstract

Evaluating the true forecasting ability of AI agents requires environments that are resistant to overtting, free from centralized trust assumptions, and grounded in incentive-compatible scoring. Existing benchmarks either rely on static datasets susceptible to training-data contamination, or measure trading prot-andloss (PnL)a metric that conates predictive accuracy with market-timing skill, position sizing, and risk appetite. We introduce Foresight Arena, the rst permissionless, on-chain benchmark for evaluating AI forecasting agents on real-world prediction markets. Agents submit probabilistic forecasts on binary markets sourced from Polymarket via a commit-reveal protocol enforced by Solidity smart contracts on Polygon PoS. Outcomes are resolved trustlessly through the Gnosis Conditional Token Framework (CTF), eliminating reliance on any centralized arbiter. Performance is measured by the Brier Score and a novel Alpha Score proper scoring rules that incentivize honest probability reporting and isolate predictive edge over market consensus, respectively. We provide a formal analysis of the statistical properties of our scoring rules: we derive closed-form expressions for the variance of per-market Alpha, establish its connection to Murphy's classical decomposition of the Brier Score into uncertainty, reliability, and resolution components, and perform a power analysis characterizing the number of rounds required to reliably distinguish agents of dierent skill levels. Applying this framework, we show that detecting a true edge of α * = 0.02 over market consensus at 80% power requires approximately 350 resolved binary predictions (50 rounds of 7 markets each), while detecting the ner edge α * = 0.01 requires roughly four times as many rounds. We complement these analytical results with a 50-round live evaluation of ve frontier LLM agents (claude-opus-4-5, gpt-5-2, gemini-3-pro, grok-4-1, glm-4-7) plus a random baseline. Murphy decomposition of the observed scores distinguishes well-calibrated agents (low reliability error) from market-tracking agents that fail primarily through reduced resolution. On-chain reputation via ERC-8004 provides agents with a veriable, persistent track record that cannot be falsied or selectively reported. All smart contracts, agent reference implementations, and evaluation infrastructure are released as open-source at https://github.com/foresight-arena/contracts.

Keywords: AI evaluation, prediction markets, proper scoring rules, Brier score, Murphy decomposition, forecasting agents

JEL Classification: C53, C45, G14, D84, C12