FinBench: Time-Gated Calibration and Uncertainty Benchmarking for Agentic Financial Forecasting

Battle Count: 74

Highly relevant to quantitative trading. The paper directly addresses the confidence-competence gap that causes capital destruction under Kelly-style bet sizing. The time-gating protocol prevents look-ahead bias, a critical concern in backtesting and live trading. The calibration metrics (Brier score, Winkler score) are directly applicable to evaluating probabilistic trading signals. The benchmark is designed for agentic systems that size trades and allocate risk, making it immediately relevant to algorithmic trading pipelines. However, the pilot is too small to draw actionable trading conclusions, and no actual trading strategy or P&L is evaluated.

Moderate complexity. The core scoring metrics (Brier score, Winkler score) are straightforward to implement. The time-gating protocol requires careful data pipeline engineering to ensure no future information leaks. The main complexity lies in: (1) building a robust time-gated data pipeline with proper timestamp handling, (2) constructing meaningful baselines (rolling base rates, momentum heuristics), (3) integrating multiple LLM APIs with structured output schemas, and (4) scaling to many tickers and trading days. The pilot itself is simple, but a production-grade benchmark requires significant infrastructure.

The paper describes a reference implementation that will be released alongside an expanded benchmark release. The pilot pipeline is described in detail (time-gating protocol, scoring formulas, baselines), but no code or data is currently available. The pilot is extremely small (33 forecasts, one trading day), limiting practical reproducibility of results. Configuration details (tickers, horizons, data windows, baselines, metric settings) are specified.