PORTBENCH: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management

Battle Count: 136

Directly benchmarks LLM-driven portfolio management across six heterogeneous asset classes with full pipeline evaluation. Provides critical evidence that financial QA capability does not translate to risk-adjusted portfolio performance (only 32.5% of 120 evaluations beat equal weighting on Sharpe). Introduces correlation-aware allocation scoring, cross-stage error propagation metrics (CEPS), stress-window evaluation, and investor-profile alignment scoring—all directly applicable to quantitative portfolio construction and risk management. The finding that models produce broad allocations far from concentrated oracle targets (mean TV distance 0.899) with only 12.8% of oracle turnover is highly relevant for practitioners considering LLM-based allocation systems.

Requires constructing a multi-asset market base dataset (183 instruments, 6 classes), implementing 7 QA task templates with closed-form ground truths, building a 5-stage stateful sandbox pipeline with deterministic execution and risk scoring, computing dual-layer correlation metrics, implementing CEPS with cascade penalties, running paired stage interventions, and evaluating across 3 stress windows × 3 profiles × 10 models. API costs for 10 frontier LLMs across multiple evaluation periods are substantial. Statistical analysis involves bootstrap methods and FDR control.

Source code publicly available at https://portbench.github.io/. Detailed evaluation protocol, QA task templates with closed-form ground truths, pipeline metrics with explicit formulas, stress window definitions, investor profile parameters, and statistical analysis procedures all fully documented. 10,000 bootstrap samples with model-cluster resampling specified. Post-processing manifest records all input/output hashes.