Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Models

Battle Count: 77

The paper is directly relevant to quantitative trading through its focus on portfolio optimization, asset allocation, and risk management (variance, Sharpe ratio, MDD, CVaR). It evaluates whether LLMs can make rational investment decisions, which is foundational for AI-driven trading systems. However, it is a benchmark/evaluation paper rather than a trading strategy paper, and does not address execution, market microstructure, or alpha generation directly.

The framework involves standard convex optimization (mean-variance portfolio), multiple constraint types, and four distractor generation methods. The mathematical foundations are well-established in portfolio theory. Implementation requires optimization solvers for generating ground truth, but the benchmark generation pipeline is modular and code is available. The main complexity lies in the systematic combination of objectives, constraints, and distractor methods to create diverse questions.

Code is publicly available on GitHub (https://github.com/noahardyx/PortBench). The benchmark is programmatically generated from portfolio theory with specified parameters (objectives, constraints, distractor methods, distance/threshold ranges). However, exact random seeds for asset selection and some parameter choices are not fully detailed in the main text. The 9,500-question dataset can be regenerated given the framework.