What survives honest evaluation? Leakage-safe, search-aware assessment of LLM-driven trading strategy discovery

Battle Count: 78

Directly addresses the core methodological problem in LLM-driven quantitative trading: reported Sharpe ratios from strategy discovery are inflated by selection bias and look-ahead contamination. Provides a complete, implementable evaluation framework (DSR, PBO, bootstrap CIs, paired tests) with structural guardrails. Demonstrates that no LLM-discovered strategy survives honest evaluation across multiple universes, cost regimes, and models, while passive benchmarks are certified. Quantifies the statistical power requirements for credible certification of moderate edges. Directly applicable to any quant team evaluating LLM-generated alpha.

High complexity: requires building a registry-validated tool surface with typed schemas, a deterministic backtesting engine with point-in-time universe selection, realistic cost models (commissions, spreads, square-root market impact, short borrow), a trial ledger system, and multiple statistical instruments (DSR, CSCV-based PBO, stationary bootstrap, paired block-bootstrap tests). The architecture involves signal graphs, portfolio archetypes, and a stateful rule engine. However, the paper provides code on Zenodo and all experiments are JSON configurations, reducing practical implementation barriers.

High reproducibility: all experiments are JSON configurations; deterministic experiments (E1, E2, E4 variants, E5 reference arms, E6, E6b) involve no LLM calls and reproduce exactly. Statistical evaluation code, experiment configurations, run manifests, and complete search traces are available on Zenodo (DOI: 10.5281/zenodo.21261868). Manifests record provider, model, code revision, package versions, and data content hashes. Raw market data from Tiingo cannot be redistributed but resumable scripts reconstruct exact panels from free-tier API key.