From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

Battle Count: 50

Highly relevant to quantitative trading. The paper directly addresses how to evaluate LLM-based trading agents in stock markets, introduces realistic A-share execution constraints (T+1, price limits, fees), provides Barra-style factor attribution that decomposes returns into market, style, and selection alpha, and demonstrates that most LLM agents' returns are explained by passive factor exposure rather than genuine stock-selection skill. The factor exposure analysis (Table 8) reveals systematic differences between ML and LLM portfolio construction. The benchmark provides a rigorous framework for distinguishing skill from luck in automated trading.

Moderate to high complexity. Requires: (1) Qlib-based execution engine with A-share market rules (T+1, price limits, board-specific thresholds); (2) Four-level masking protocol with alias maps and tool-level re-masking; (3) Barra-style cross-sectional regression with VIF-screened factors; (4) ReAct-style agent loop with six tools and JSON action schema; (5) Multi-seed evaluation infrastructure; (6) De-anonymization probe framework. The masking protocol and attribution layer are the most novel components. Vendor API integration and provider-neutral adapters add engineering complexity.

The authors explicitly state they release KTD-FIN as a reproducible template including environment, scripts, masking and probe protocols, and the full evaluation dataset. The harness is deterministic given a seed, and all models are re-evaluable on identical conditions. Provider-neutral tool-call normalization is implemented. Full hyperparameters, prompts, and configuration tables are provided in appendices.