The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysis

Battle Count: 50

Highly relevant for LLM-based trading systems and financial decision support. The paper demonstrates that role-conditioned bias can flip BUY/SELL recommendations on the same filing, which directly impacts algorithmic trading signals derived from LLM analysis. The finding that within-persona rankings retain predictive signal (Q5−Q1 CAR of +2.96% panel mean) while levels shift with persona conditioning is critical for signal extraction. The audit protocol provides a practical evaluation framework for any LLM deployed in trading pipelines. However, the paper does not propose a trading strategy itself; it focuses on reliability and invariance testing.

The audit design is conceptually clear but operationally involved: requires building per-filing retrieval indices, running 12 models × 10 personas × 3 conditions × 3,575 filings, implementing filing-fixed-effects regressions with bootstrap CIs, and computing multiple derived metrics. The prompt templates and output schemas are fully specified. No fine-tuning is required (inference-only). The main complexity lies in the scale of experimentation and the statistical analysis pipeline rather than novel algorithmic components.

The paper provides detailed prompt templates (Appendix C), exact retrieval queries, output schemas, regression specifications, and sample construction. All 12 models are listed with deterministic decoding parameters (T=0, top-p=1). However, no GitHub repository is mentioned, and the SEC filing sample construction details (exact accession numbers) are not fully enumerated. The cross-domain checks cover only 2 of 12 models.