All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection

Battle Count: 50

The paper addresses financial misinformation detection, which is relevant to quantitative trading as misleading financial narratives can affect market sentiment and trading decisions. However, the paper focuses on LLM evaluation for misinformation detection rather than direct trading strategy development, risk management, or portfolio optimization. The findings about LLMs' inability to detect counterfactual perturbations without external grounding are relevant for systems that use LLMs to parse financial news for trading signals.

The benchmark construction pipeline involves multiple stages: data acquisition from Yahoo Finance, rule-based keyword classification, GPT-4.1 category-specific rewriting with constrained decoding, automatic quality control (token-length ratios), expert review, dual-annotator evaluation, and structured adjudication. The evaluation itself is straightforward (prompting LLMs with paragraphs), but reproducing the full dataset construction requires domain experts, multiple annotators, and API access to GPT-4.1. The four manipulation categories require different prompt engineering and validation approaches.

Dataset released on GitHub with structured metadata. All prompts (rewriting, evaluation, few-shot) provided in appendices. Detailed annotation guidelines, expert review procedures, and metric definitions included. However, the LLM-based rewriting pipeline depends on specific GPT-4.1 API access and decoding parameters. Human annotation process is well-documented but requires domain experts.