Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy

Battle Count: 50

The paper includes a dedicated financial experiment (Appendix H) generating S&P 500 log-returns (d=424, K=2766). SA reproduces return unpredictability (near-zero autocorrelation) and cross-sectional dependence structure (Frobenius correlation error 26.3%), and generates novel regime interpolations (novelty 0.768) absent from historical data. However, it fails to reproduce volatility clustering (a non-stationary phenomenon), which is a fundamental limitation of equilibrium Boltzmann sampling. The method could serve as a training-free scenario generator for risk management in low-data regimes, but cannot capture temporal dynamics like volatility clustering without extensions.

The core algorithm (Algorithm 1) is extremely simple: each step requires two matrix-vector products, one softmax, and one Gaussian draw at cost O(dK). No training loop, no score network, no contrastive objective. The temperature β is the only hyperparameter (plus step size α). The entropy inflection criterion provides a principled, data-dependent β selection. However, practical considerations include: multi-chain initialization for cross-basin coverage at high β, burn-in and thinning for MCMC diagnostics, PCA preprocessing for high-dimensional data, and MALA variant for larger step sizes. The masking extension for conditional generation is a one-line change.

The paper provides Algorithm 1 (SA sampler) and Algorithm 2 (MALA variant) in full pseudocode. Hyperparameters are explicitly stated (α=0.01, β values, T=5000, burn-in=2000, thinning=100). Baseline specifications (VAE architecture, DDPM schedule, GMM-PCA parameters) are detailed in appendices. Code is referenced as publicly available in code/mnist-experiment/ and code/vae-experiment/ in the accompanying repository. Fixed random seeds are used. The method is inherently reproducible due to its training-free nature and closed-form score function.