Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements

Battle Count: 70

The paper focuses on financial document question answering and regulatory compliance rather than quantitative trading strategies. While the extracted financial indicators (CET1, Total Assets, liquidity reserves) could inform fundamental analysis or risk management inputs for trading systems, the paper does not address trading signals, portfolio construction, or market prediction. The primary contribution is a benchmark dataset and RAG pipeline evaluation methodology.

The multi-stage RAG pipeline involves: PDF processing with Azure Document Intelligence, custom hierarchical chunking, contextual enrichment via GPT-4.1 (with dynamic heading-level adjustment for large files), vector indexing in Qdrant, optional cross-encoder reranking, and LLM generation. Integration of multiple proprietary APIs, handling of very long documents (avg 198k words), and managing the contextual enrichment process for files exceeding model context windows add significant engineering complexity. However, the individual components are well-documented and modular.

The benchmark dataset is publicly available on GitHub. However, the pipeline relies on proprietary APIs (OpenAI GPT-4o, GPT-o1-high, VoyageAI, Cohere, Azure Document Intelligence) which may change over time. The stochastic nature of reasoning models (GPT-o1-high) affects result reproducibility. Experiments were conducted in late 2024–early 2025 with models available at that time, limiting temporal reproducibility.