Can Open-Weight Models Compete on Financial Text Comprehension?

Battle Count: 62

The paper is highly relevant to quantitative trading insofar as it evaluates LLMs' ability to extract and comprehend financial information from annual reports, which are critical inputs for fundamental analysis, earnings prediction, and investment decisions. However, it does not directly address trading strategies, market microstructure, or quantitative signal generation. The findings on model accuracy, hallucination rates, and retrieval bottlenecks are relevant for building reliable financial information extraction pipelines that feed into trading systems. The content filter findings are particularly relevant for global trading desks using Chinese AI models.

The RAG pipeline is intentionally simple (FAISS + text-embedding-3-small + top-5 retrieval), making it straightforward to reproduce. However, the full evaluation requires access to 20 different model APIs from 10 providers, LLM-as-a-judge grading with specific instructions, and the 495-report dataset. The contamination checks and error taxonomy analysis add methodological complexity. The content filter investigation requires testing multiple access routes.

Complete dataset and evaluation framework are publicly available at financial-touchstone.datascience-nlp.ai/. The paper provides detailed model configurations (Table 3), retrieval pipeline specifications, judge instructions, and contamination checks. All 20 models, routes, and API identifiers are documented. The RAG pipeline is intentionally simple and reproducible. Inter-annotator agreement procedures are documented.