When Reasoning Fails: Evaluating "Thinking" LLMs for Stock Prediction

Battle Count: 150

Highly relevant to quantitative trading. The paper directly addresses whether 'thinking' LLMs can improve short-horizon cross-sectional stock ranking, a core task in equity quant strategies. It provides rigorous walk-forward evaluation, realistic Indian market cost modeling (STT, brokerage, slippage, borrow), portfolio backtests with turnover caps, and formal statistical tests (DM, PT, SPA). The findings—that TLLMs underperform classical baselines under fixed budgets and that next-token objectives misalign with financial prediction—are directly actionable for practitioners considering LLM-based signals. The practical guidance (use direct LLMs as auxiliary features, not sole forecasters) is immediately applicable.

Moderate complexity. The pipeline involves: (1) feature engineering from daily OHLCV (momentum, volatility, RSI, MACD, cross-sectional ranks, market context), (2) LLM API calls with structured prompts and TSV output parsing, (3) per-date calibration (z-score, rescale, winsorize), (4) walk-forward evaluation with rolling windows, (5) portfolio construction with cost modeling, and (6) multiple statistical tests (DM, PT, SPA). The LLM integration is straightforward via API, but the full evaluation harness with proper leakage controls, cost modeling, and statistical testing requires careful implementation. No public code is provided, so reproduction requires building the pipeline from scratch.

The paper specifies fixed seeds, per-date feature snapshots, model-output caches, and a configuration manifest for exact reproduction. Walk-forward protocol, calibration pipeline (z-score→calibrate→winsorize), and cost model are fully described. However, no public code repository is explicitly linked, and the exact NIFTY constituent list is referenced only via a configuration manifest. The system prompt is provided verbatim. Some tables reference 'Table??' and 'Fig.??' suggesting incomplete cross-references in the draft.