FinReflectKG - EvalBench: Benchmarking Financial KG with Multi-Dimensional Evaluation

Battle Count: 64

The paper is primarily about evaluation methodology for financial knowledge graph extraction rather than direct trading strategy development. However, the extracted KGs from SEC filings can support downstream quantitative applications such as risk signal generation, compliance monitoring, and investment research. The benchmark ensures quality of structured financial data that feeds into trading and risk management pipelines. The work is more foundational/infrastructure-oriented than directly applicable to alpha generation or portfolio optimization.

Implementation requires: (1) a deterministic structure-aware chunking scheme for SEC filings, (2) three different extraction pipelines (single-pass, multi-pass, reflection-based agentic workflow), (3) a heterogeneous LLM-as-Judge ensemble with three different models configured for deterministic decoding, (4) carefully designed prompts with bias controls and few-shot examples, (5) commit-then-justify output parsing, and (6) Krippendorff's alpha computation. The multi-model ensemble and agentic reflection loop add significant engineering complexity, though the core evaluation logic is well-specified.

The paper describes a detailed evaluation protocol with deterministic decoding (temperature=0.0), explicit bias controls, and concrete prompt examples in the appendix. However, no code repository or dataset link is explicitly provided. The use of specific LLM judges (QWEN3, GPT-OSS, LLAMA) and SEC 10-K filings (publicly available) aids reproducibility, but the exact prompts and chunking scheme details may require additional documentation.