RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?

Battle Count: 84

While not directly about trading strategies or market prediction, the paper is highly relevant to quantitative trading infrastructure. It reveals critical failure modes in LLMs used for financial decision-making: over-commitment when information is missing, inability to identify under-specified problems, and false confidence. These findings directly impact the reliability of LLM-based trading agents, risk assessment tools, and portfolio management systems. The bilingual evaluation (English CFA and Chinese CPA) is relevant for global quantitative trading firms. The NOTA evaluation framework could inform guardrails for automated trading systems.

The benchmark itself is straightforward to implement: it involves evaluating existing models on a curated dataset with structured JSON output prompts. No model training or modification is required. However, reproducing the full evaluation requires access to 15 different models (some via API, some requiring local GPU deployment with specific hardware like NVIDIA H20/H100/RTX 4090), and the GPT-OSS models require specialized software environments (Transformers >=4.55, Triton >=3.4). The dataset curation process involves manual annotation with quality control.

Dataset and code are publicly available at https://github.com/insait-institute/RealFin. All 15 models are evaluated with fixed temperature (0), greedy decoding, and detailed hardware/software configurations provided in appendices. Prompts are fully specified for both English and Chinese. The dataset construction process is formally defined with logical constraints.