Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books

By Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh, George Nigmatulin, Kang Li, Stefan Zohren, Jakob Foerster, Mihai Cucuringu, Anisoara Calinescu

Rating

1769
Battle Count: 50

Relevance

9/10
Highly relevant for understanding the internal representations of modern generative market models. It provides insights into how foundation models encode order-flow imbalance, a key alpha signal, and offers a method for interpreting and potentially steering these models for better simulation or prediction.

Implementation Complexity

7/10
Requires implementing or accessing a Mamba-3 architecture, handling high-frequency LOB data tokenization, and performing advanced interpretability techniques like kernel ridge probing on high-dimensional recurrent states and causal steering interventions.

Reproducibility

3/5
The paper provides detailed architectural specifications (Mamba-3, 8.2M params), training data description (LOBSTER format, specific tickers, dates), and probing/steering methodologies. However, no public code repository URL is explicitly provided in the text extract, and the specific LOBSTER data access might require a subscription.

Abstract

Autoregressive foundation models offer state-of-the-art capabilities to simulate the market by generating the message flow of limit order books (LOBs). Yet, without understanding their inner workings, research directions are hard to narrow and practical use in real-world trading environments remains uncertain. This study therefore asks what these models actually learn about market microstructure. We find that, similarly to games where sequence models build an internal representation of the board, an 8.2M-parameter Mamba-3 state-space model trained autoregressively on the tokenized message stream of eight large-cap US equities learns an internal representation of the market and its dynamics. In particular, order-flow imbalance, a central driver of short-horizon price moves, is decodable at R^2 ≈ 0.59, far above controls and baselines. This representation extends to unseen stocks, suggesting that these models learn general representations rather than ticker-specific patterns. The model also generalizes across metrics beyond order-flow imbalance: results hold across traded volume and volatility. Additionally, the representation is causally steerable: pushing the state along a feature's axis shifts the model's next-message beliefs as that feature implies, monotonically in dose, consistent with the emergence of a world model.

Links

arXiv:2610.28753 · Paper rankings

The interactive Everscope explorer (charts, battles, favorites) loads below.