Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA

Battle Count: 50

The paper is primarily about agentic QA auditing in financial due diligence and regulated document work, not about trading strategy development or market prediction. However, it is relevant to quantitative trading insofar as it addresses the reliability of LLM-based agents performing multi-hop document analysis (credit memos, mandates, NAV reports, sanctions lists) that inform investment decisions. The findings about buried evidence degradation, confident wrong answers, and cost escalation are directly applicable to any trading desk deploying agentic tools for research. The paper does not address price forecasting, portfolio optimization, or execution.

Moderate complexity. Requires setting up a synthetic data room with known provenance, configuring agent harnesses with tool access (bash, python, read, grep, find), running models under clean and buried conditions with forced-declaration JSON schemas, implementing a containment-based grader, and conducting structured human audits. The frozen evidence folder, claim ledger, and hash manifest infrastructure add operational overhead. No model training is involved; this is an evaluation/audit framework. Reproducing the exact results requires access to the specific model APIs and the frozen corpus.

Extremely high reproducibility: frozen evidence folders with SHA-256 hash manifests (64 files), a 420-row claim ledger mapping every numeric claim to a file and locator, byte-identical table regeneration verified on independent re-run, Zenodo deposit (DOI 10.5281/zenodo.22310532) under CC-BY-4.0, frozen pricing snapshot, and a structured grader audit with 94 recorded human decisions. Mechanical re-verification of all 420 ledger rows confirmed 420/420 match. Independent human sign-off of rows is still pending.