AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models

Battle Count: 81

Highly relevant to quantitative trading. The paper directly addresses the fundamental problem of LLM instability in trading decision-making and proposes a principled solution by reframing LLMs as quantitative researchers generating executable strategy code. The benchmark evaluates alpha factor discovery, factor-based strategy composition, and end-to-end trading strategy design - all core activities in quantitative finance. The 3x3 difficulty taxonomy provides diagnostic insights into model capabilities relevant to real-world quant workflows. The backtesting framework across crypto and equity markets with standard financial metrics (SR, ARR, MDD, CR, SoR, VOL) directly mirrors industry evaluation practices. The finding that different LLMs encode stable 'risk personalities' has practical implications for model selection in trading systems.

Moderate to high complexity. The framework requires: (1) a unified backtest engine with standardized execution protocol, (2) a comprehensive factor library with 120+ precomputed technical indicators, (3) a two-stage dataset construction pipeline involving LLM-based extraction and generation, (4) multi-asset backtesting infrastructure spanning crypto and equity markets, (5) statistical analysis protocols with multi-run variance decomposition, and (6) API integration with multiple LLM providers. However, the core evaluation logic (prompt → code generation → deterministic backtest) is conceptually straightforward. The paper provides detailed prompts, factor definitions, and parameter specifications in appendices.

The paper provides a comprehensive evaluation framework with deterministic backtesting, standardized prompts, and detailed experimental settings. All code and data are stated to be publicly available upon acceptance. The benchmark uses 5 independent runs per query with mean±std reporting. Temperature ablation (T=0 vs T=0.7) demonstrates near-perfect invariance. However, the backtest engine code and full dataset are not yet publicly released (pending acceptance). The paper provides detailed prompts, factor library specifications, and backtest parameters in appendices.