More Mentions, Less Information? Diagnosing Identity and Alignment in Reddit Embeddings for Volatility Forecasting

By Zhiyu Zheng

Rating

1829

Battle Count: 50

Relevance

8/10

Highly relevant for quantitative researchers using alternative data (social media) for volatility forecasting. It provides a rigorous framework to avoid overestimating the value of text embeddings by controlling for firm identity and alignment.

Implementation Complexity

7/10

Requires handling large-scale text data, generating LLM embeddings, implementing rolling out-of-sample validation, and performing complex bootstrap and permutation tests.

Reproducibility

2/5

Code is not publicly released at the time of writing but is available from the author on request. Data (Reddit Pushshift, TAQ, CRSP) requires specific licenses or archives that may not be easily accessible.

Abstract

Since the 2021 GameStop short squeeze, r/WallStreetBets (WSB) has been associated with episodes of extreme volatility, and its text is increasingly used to forecast volatility. A model using only large language model (LLM) embeddings of WSB posts attains an out-of-sample R^2 of 0.13 for log realized variance. That number shows predictive power, but it does not reveal where the information comes from or how much the text adds beyond a firm's own volatility history. We answer these questions with a diagnostic protocol that nests embeddings on a heterogeneous autoregressive (HAR) benchmark, compares them with text-free firm-identity controls, shuffles the day-to-text pairing within firm-years to test day-specific alignment, and checks whether gains persist across periods and coverage groups. We apply the protocol to frozen LLM embeddings in a panel of 202,145 mentioned stock-days, using retrospective rolling out-of-sample tests over 2015-2022. The results indicate that most of the standalone fit reflects firm or firm-year information: on firms seen in training, a text-free firm baseline reaches R^2 = 0.32, shuffling preserves 89-92% of the standalone fit, and masking ticker strings reduces standalone R^2 to -0.005. Adding embeddings to HAR reduces equal-date MSE by 1.22% (95% CI [0.94, 1.49]). This small but statistically significant gain survives firm controls and ticker masking, but is largely eliminated by alignment shuffling. The gain is concentrated in 2015-2019 and is not distinguishable from zero in 2020-2022. Expanding coverage by 48% through off-hours and weekend posts adds stock-days with no reliable positive increment. More mentions therefore do not imply more incremental forecasting information.

Links

arXiv:2610.25468 · Paper rankings

The interactive Everscope explorer (charts, battles, favorites) loads below.