PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data

Battle Count: 72

Highly relevant to quantitative trading. The paper directly evaluates LLMs as trading agents on live prediction markets with realistic CLOB execution simulation, order-book slippage modeling, position sizing effects, and financial return metrics (CWR, APY, Sharpe Ratio). It demonstrates that language fluency does not guarantee profitable trading, highlights the importance of risk calibration and domain-specific competence, and shows how market microstructure (liquidity constraints, spread widening) erodes theoretical alpha. The findings on miscalibrated confidence across domains (Politics vs. Crypto) are directly applicable to trading strategy design.

Moderate complexity. The benchmark construction pipeline involves API integration with Polymarket Gamma API, Google News fetching, CLOB state capture, and structured prompt engineering. The evaluation framework requires order-book execution simulation with liquidity absorption logic. However, the code is publicly available, and the core methodology relies on calling existing LLM APIs with structured prompts and parsing JSON outputs. The main complexity lies in the data collection infrastructure and the realistic execution simulation logic.

Dataset and code are publicly available at https://github.com/PolyBench/PolyBench. The benchmark uses live Polymarket data collected between Feb 6-12, 2026, with structured JSON output schemas and detailed prompt design documented. However, the live nature of the data means exact reproduction depends on the archived snapshots. Model API costs and quantization details are provided. The evaluation protocol is well-specified with clear thresholds (confidence > 0.6) and execution simulation rules.