Momentum, Commentary Tone, and Earnings Surprise in LLM Financial Forecasts

By Alina Khaybullina

Rating

1702
Battle Count: 50

Relevance

7/10
Highly relevant for understanding how LLMs process financial signals (momentum, tone, surprise) before deployment in trading strategies. It highlights the fragility of LLM forecasts to prompt formatting and quantization, which is critical for robust algorithmic trading systems.

Implementation Complexity

6/10
Requires setting up llama-cpp-python with specific pinned revisions, handling GGUF conversions, and implementing a complex factorial design with multiple answer encodings and controls. The analysis involves bootstrap resampling and surrogate modeling.

Reproducibility

5/5
Public benchmark prompts, recorded scores, and an interactive recorded-results explorer are provided. The dataset is available on Hugging Face, and a GitHub showcase is linked. The methodology is fully described with pinned revisions for models and tokenizers.

Abstract

Which inputs drive a language model's financial forecast? We construct synthetic earnings announcements that independently reverse 20-day momentum, non-informational commentary tone, and whether fixed actual EPS beats a manipulated consensus. Six instruction/chat checkpoint conversions from four families (0.49-7.61B base parameters) receive 2,016 prompts each under a direct-answer Q8_0 protocol. The design includes 24 numerical blocks, six commentary pairs, profits and losses, rising and falling annual EPS, and two surprise sizes. Both A/B answer assignments and U/D cover all 384 primary scenarios per conversion. We estimate matched changes in conditional answer-prefix scores, not calibrated market probabilities. Across these checkpoints, the primary EPS contrast ranges from -0.19 to 86.80 percentage points. The magnitude of the primary signed tone contrast exceeds that of EPS in 5 of six models, and momentum exceeds it in 5. Natural-word numerical diagnostic accuracy ranges from 51.0 to 95.8 percent. Answer-encoding, wording, context, and conversion controls qualify these results. The experiment measures observable input reliance and numerical recognition; it establishes neither an internal reasoning mechanism nor realized return predictability. Public benchmark prompts, recorded scores, and an interactive recorded-results explorer accompany the study.

Links

arXiv:2610.22043 · Code · Paper rankings

The interactive Everscope explorer (charts, battles, favorites) loads below.