FinReflectKG - MultiHop: Financial QA Benchmark for Reasoning with Knowledge Graph Evidence

Battle Count: 67

The paper is primarily about financial QA benchmarking and knowledge graph retrieval rather than direct trading strategy development. However, it has indirect relevance: (1) efficient multi-hop reasoning over SEC filings could support fundamental analysis and signal generation; (2) the finding that KG-guided retrieval reduces token costs by ~84.5% while improving accuracy by ~24% is relevant for building cost-efficient NLP pipelines in trading firms; (3) understanding temporal and cross-company financial relationships supports risk assessment and factor analysis; (4) the benchmark could be used to evaluate LLM-based research assistants for quantitative analysts. The connection to actual trading decisions, portfolio optimization, or market prediction is indirect.

High complexity due to: (1) requires constructing or accessing a temporally indexed financial knowledge graph with 17.5M triplets; (2) multi-phase pipeline involving pattern generation via LLM, Cypher query validation, graph traversal, betweenness centrality computation; (3) quality control with multi-criteria scoring rubrics; (4) three controlled evidence retrieval scenarios requiring careful chunk identification and windowing; (5) evaluation across multiple model families with fixed prompts; (6) dual LLM-as-Judge evaluation. However, the released 555 QA pairs lower the barrier for using the benchmark itself.

The authors release a curated subset of 555 QA pairs and provide detailed experimental protocols including fixed prompts and decoding parameters. However, the full benchmark is still under manual expert validation, the KG construction pipeline (FinReflectKG) is referenced but not fully open-sourced in this paper, and evaluation uses proprietary Gemini 2.5-Pro as an alternative judge. The evaluation subset is limited to 150 QA pairs across 2 GICS sectors. Private cloud deployment for experiments may limit exact reproducibility.