Seeing the Goal, Missing the Truth: Human Accountability for AI Bias

Battle Count: 79

Highly relevant to quantitative trading. The paper directly addresses a critical methodological concern for quants using LLM-generated sentiment or text-based signals in trading strategies: disclosing the downstream prediction objective (e.g., stock returns) to the LLM inflates in-sample performance but fails to generalize out-of-sample. This has direct implications for backtesting, signal construction, and live deployment of LLM-based alpha factors. The portfolio-sorting and Fama-MacBeth regression frameworks are standard in quantitative finance. The finding that prompt-level regularization cannot fully fix the bias is particularly important for practitioners who rely on careful prompt engineering. The knowledge-cutoff-based natural experiment provides a clean methodology for validating LLM-derived signals.

Moderate complexity. The core experiment requires: (1) access to LLM APIs (GPT-4o-mini, Gemini 2.5) with well-defined prompts, (2) proprietary financial data (Capital IQ transcripts, CRSP, Compustat), (3) standard econometric implementations (portfolio sorts, Fama-MacBeth regressions, expanding-window OOS forecasts). The prompt design is straightforward (single-sentence variation), but the full replication requires significant financial data infrastructure. The reasoning-trace analysis with GPT-5 and the conversation-history experiment add additional layers. No specialized ML training or fine-tuning is needed; the study is purely inference-time.

Full prompts are provided in Appendix B, and the experimental design is clearly described. However, the underlying data (Capital IQ earnings call transcripts, CRSP stock returns, Compustat accounting data) are proprietary and not publicly available. The LLM API calls (GPT-4o-mini, Gemini 2.5, GPT-5) are reproducible in principle but may yield slightly different outputs over time due to model updates. No code or data repository is mentioned. The knowledge cutoff dates are well-documented, enabling replication of the pre/post-cutoff analysis structure.