FinSkillBench: Evaluating AI Agents and Domain Skills for Investment Management

Battle Count: 56

Highly relevant to quantitative trading and investment management. The benchmark directly evaluates agents on portfolio construction (mean-variance optimization, Black-Litterman, constrained optimization, rebalancing), risk management (mandate monitoring, stress testing, risk identification, remediation trades), and fundamental analysis (XBRL normalization, earnings quality via Piotroski/Beneish, driver decomposition). These are core workflows in institutional quantitative finance. The findings that curated skills provide +16.2pp improvement while self-generated skills provide negligible benefit (+0.5pp) have direct implications for building production financial agents. The point-in-time discipline and structured output requirements mirror real trading system constraints. However, the paper is primarily a benchmark/evaluation paper rather than proposing new trading strategies or models.

Moderate to high complexity. The benchmark requires: (1) setting up a ReAct-style agent loop with function calling, (2) implementing 12 subtask-specific verifiers with domain-specific scoring logic, (3) building data pipelines for portfolio construction (Ledoit-Wolf covariance, factor exposures, convex optimization via cvxpy), risk management (synthetic portfolios, stress scenarios, risk engines), and fundamental analysis (XBRL parsing, Piotroski/Beneish computation), (4) creating curated skill packages with both declarative Markdown and imperative Python scripts, (5) managing point-in-time data discipline with lossy compaction, and (6) running 17,820+ episodes across 9 models and 3 conditions. The cross-harness validation adds additional complexity. However, the authors release all tools and trajectories.

The authors release the benchmark, evaluation tools, curated skill packages, and full trajectories. Ground truth is deterministic (convex solvers, Black-Litterman routines). However, time-dependent provider APIs, model version changes, and tool-calling reliability issues (Phi-4, grok-4 failures) limit full external reproducibility. Single-sample cells with temperature 0 reduce but do not eliminate variance. Cross-harness validation on Hermes Agent adds robustness. The 30-ticker universe is fixed and frozen.