PredictionMarketBench: A SWE-bench-Style Framework for Backtesting Trading Agents on Prediction Markets

Battle Count: 78

Directly relevant to quantitative trading as it provides a standardized, execution-realistic backtesting framework for trading agents. The emphasis on maker/taker semantics, fee modeling, and deterministic replay addresses critical practical concerns in algorithmic trading. The finding that transaction costs and settlement losses can dominate strategy performance is highly relevant. However, it is specifically focused on prediction markets (binary contracts) rather than traditional equities/futures, and the current dataset is very small.

Moderate complexity: requires understanding of limit order book mechanics, maker/taker execution semantics, fee modeling for binary contracts, event-driven simulation architecture, and agent interface design. The episode format is straightforward (parquet files + JSON), but building a faithful execution simulator with queue priority, partial fills, and settlement logic requires careful implementation. The agent interface is lightweight (Kalshi-like API primitives).

Highly reproducible: deterministic event-driven replay with fixed seeds, standardized episode format (self-contained directories with metadata, orderbook.parquet, trades.parquet, settlement.json), fixed simulator configuration, full trajectory logging with checkpoints every 50 steps, and code released. Same episode files + simulator config + agent code with fixed seeds guarantee identical runs.