Leakage-Aware Benchmarking of LLM Forecasting: Real-Time Nowcasts as the Decision-Time Input for Macro Factor Ranking

Battle Count: 82

Directly addresses equity style factor ranking (SMB, HML, RMW, CMA, UMD, BAB, QMJ) under realistic decision-time constraints. The leakage-aware benchmarking methodology is highly relevant for practitioners building factor-timing models. The finding that real-time inflation nowcasts and macro-similar retrieval explain much of the median signal, while LLM adds value in extreme rankings driving long-short portfolios, has direct implications for quantitative strategy design. However, the underpowered results (n=36, p=0.11) limit immediate practical deployment claims.

The pipeline uses a 7B LLM with 4-bit quantization runnable on commodity GPU hardware, which lowers the barrier. However, the full system requires: (1) retrieval-augmented generation with cosine similarity on z-scored macro state vectors, (2) two-stage LLM inference (critic then actor), (3) rolling rule buffer management, (4) strict decision-time data pipeline with lag-shifting and nowcast integration, (5) expanding-window standardization, and (6) 12-month embargo retrieval. The data engineering for leakage-controlled inputs is non-trivial.

Authors commit to releasing walk-forward harness, all prompt templates, per-month score parquets, parsed Cleveland Fed nowcast archive (frozen 2026-05-13), retrieved analog indices, full critic-rule log, and exact Qwen2.5-7B-Instruct 4-bit inference configuration. Bootstrap and permutation tests use np.random.seed(42). However, the paper is non-archival (ICML 2026 Workshop) and the release is stated as future ('We will release'), so full reproducibility depends on actual publication of artifacts.