Reasoning Models Ace the CFA Exams

Battle Count: 71

The paper is primarily about evaluating LLM capabilities on financial knowledge and reasoning tasks rather than developing trading strategies. However, the findings are relevant to quantitative trading in several ways: (1) demonstrates LLMs can handle complex financial calculations, portfolio construction, and risk analysis; (2) identifies persistent weaknesses in ethical judgment and nuanced financial reasoning; (3) the CFA curriculum covers topics directly relevant to quantitative finance including derivatives, fixed income, equity valuation, and portfolio management; (4) findings inform the feasibility of using LLMs as decision-support tools in trading environments.

The methodology is relatively straightforward: API calls to various LLM providers with structured prompts, temperature=0, and automated scoring. The main complexity lies in compiling the mock exam dataset from proprietary sources and setting up the automated CRQ grading pipeline. No model training or fine-tuning is involved.

The paper provides specific model identifiers and snapshot dates (Table 4), temperature settings (0), exact prompt templates for both ZS and CoT conditions (Section A), pass/fail criteria, and data sources. However, the mock exam datasets are proprietary (CFA Institute Practice Pack and AnalystPrep), limiting full reproducibility. Results are reported as mean ± standard deviation across runs.