FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings

Battle Count: 50

The paper explicitly states it is not designed for automated investment decisions, valuation, or trading signals. However, it is relevant to the broader financial NLP ecosystem: accurate evidence-grounded retrieval over SEC filings could support fundamental analysis workflows that inform quantitative strategies. The benchmark's focus on distinguishing correct evidence from confusable distractors (competitor filings, prior periods) addresses a practical challenge in building reliable financial data pipelines. The paper is primarily an NLP/IR benchmark contribution rather than a quantitative trading methodology.

The benchmark itself is straightforward to use: a single .jsonl file with a well-documented schema, a released evaluation harness with scripts that complete in under 5 minutes on a single A10G GPU (or ~1 hour on CPU for smaller models). The 7B e5-mistral model requires GPU. The normalization pipeline is deterministic and idempotent. However, building a competitive system on FinRank requires understanding the two evaluation regimes (global pooled corpus C vs. in-record candidate set Lr), the hard-negative taxonomy, and stratified reporting. The benchmark design is conceptually rich but the evaluation infrastructure is well-packaged.

The paper provides exceptional reproducibility: the dataset is released as a single .jsonl file with a deterministic label-normalization pipeline, a machine-readable repair log, per-passage identifiers and text hashes, a hard-negative taxonomy (hntaxonomy.json), and a complete evaluation harness in baselines/ with exact scripts, model versions, seeds (seed=42), and runtime expectations. Five generalization splits are released as record-ID assignments. The full question-generation methodology is reproduced in Appendix A. All reported numbers are emitted by baselines/make_macros.py and are exactly reproducible from the released artifacts. The GitHub repository includes dataset, evaluation harness, repair log, and hard-negative taxonomy.