CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents

Battle Count: 50

Directly addresses the evaluation of LLM-based portfolio management agents in a closed-loop trading context. Introduces realistic multi-component transaction cost modeling, point-in-time data enforcement, strategy consistency measurement, and a five-axis capability diagnostic framework. Validates against passive baselines (SPY, IEF, 60/40, equal-weight, risk-parity, momentum) and includes live broker paper-trading. Key findings (stating-vs-doing gap, capability ≠ outcome, module value invisible to returns) are directly actionable for quantitative trading practitioners evaluating AI agents. The instrument is designed as a pre-deployment gate for LLM traders.

Extremely complex system requiring: five-stage closed-loop decision engine with six specialized agents, TimeGate temporal enforcement, four parameterized cost models across asset classes and tiers, three-tier memory with decay and consolidation, 19 MCP-defined tools across four namespaces, recompute-verifiable hash chain, mandate-aware portfolio construction with constrained projection, 3-D SCOUT stage, intertemporal sentiment trajectory tracking, held-out LLM-as-judge protocol, structured vs. autonomous skill modes, live broker paper-trading integration, 13-configuration ablation grid, and five-axis capability scorecard computation. Requires deep integration of LLM APIs, quantitative finance libraries, broker APIs, and audit infrastructure.

Every number recomputes from persisted per-run audit stores. Hash-chain verification enables independent integrity checks. Campaign configuration is declarative (TOML). Judge scores are cached by round_id. Model identities and snapshots pinned via OpenRouter. Data providers named. Audit databases, scorecard/analysis tooling, and figure-generation scripts available from corresponding author on reasonable request. However, no public GitHub repository is explicitly linked, and some artifacts require request.