StakeBench: Evaluating Language Understanding Grounded in Market Commitment

Battle Count: 50

StakeBench is highly relevant to quantitative trading as it evaluates whether LLMs can understand market commitment signals embedded in language, anticipate trading actions, and project collective odds movements. The G4 task directly tests odds-direction prediction beyond naive baselines. However, the paper primarily focuses on evaluation/benchmarking rather than proposing trading strategies. Key findings (models fail on future action anticipation and collective odds projection, finance-domain tuning does not help, platform incentives shape results) have direct implications for using LLMs in trading signal extraction. The revealed-preference framework and commitment-aware metrics could inform feature engineering for NLP-based trading systems.

The benchmark itself is well-documented with code under CC-BY 4.0. Implementation requires: (1) API access to Polymarket and Manifold for data collection, (2) position reconstruction via trade history replay, (3) evaluation across 15 models with fixed prompting, (4) statistical testing (binomial, Mann-Whitney, BH-FDR). The data pipeline involves parallelized API requests with exponential backoff. Model evaluation uses standard Hugging Face Transformers with greedy decoding. The main complexity lies in the data collection and position reconstruction pipeline rather than the evaluation metrics themselves.

StakeBench is packaged with evaluation code and dataset under CC-BY 4.0. All data retrieved from public APIs without authentication. Fixed prompting protocol with greedy decoding (temperature 0, seed 42). Full prompt templates, API endpoints, collection funnel, filtering parameters, and statistical details provided in appendices. 15 models evaluated on same 18 topic-platform splits. Macro averaging over splits prevents high-volume market dominance.