Semantic Faithfulness and Entropy Production Measures to Tame Your LLM Demons and Manage Hallucinations

Battle Count: 69

The paper is primarily about LLM evaluation and hallucination detection, not quantitative trading. However, it has indirect relevance: (1) the author is affiliated with Fidelity Investments, suggesting financial domain motivation; (2) the framework could be applied to evaluate LLM-generated financial summaries of SEC filings, earnings reports, or market analyses used in trading workflows; (3) hallucination detection in financial document summarization is critical for trading decisions; (4) the reference-free evaluation approach could help validate LLM outputs in financial NLP pipelines. However, the paper does not address any trading-specific problems, market prediction, portfolio optimization, or risk management directly.

Moderate complexity. The core algorithms (Alternating Minimization for SF, dual Lagrangian optimization for SEP) are well-specified and use standard convex optimization (scipy.optimize, L-BFGS-B). The pipeline requires: (1) sentence embedding with a pre-trained model (Qwen3-Embedding-0.6B), (2) topic clustering via UDIB, (3) computing marginal distributions, (4) running the AM algorithm (12-69 iterations, median 18). The mathematical formulation involves KL divergence, Lagrangian optimization, and fixed-point iterations, but the actual implementation leverages existing libraries. The main complexity lies in the topic modeling/clustering step and ensuring proper convergence. Code is provided on GitHub, reducing implementation burden significantly.

Code and data are publicly available on GitHub (https://github.com/ighalp/semantic-faithfulness-sdm). The methodology uses standard Python scientific stack (scipy.optimize) and a publicly available sentence embedding model (Qwen3-Embedding-0.6B). Algorithms are fully specified with pseudocode (Algorithm 1 and 2). However, the experimental dataset is small (10 QCA triplets from a single source document), and the LLM-as-a-Judge evaluation uses a specific model (Claude Sonnet 4.5) which may introduce variability. The paper notes that SF values can vary across experimental sessions due to different random initializations.