Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Series

Battle Count: 66

Highly relevant to quantitative trading. The paper directly addresses stock return prediction using multimodal data (news + price history), demonstrates state-of-the-art AUC performance (57.94% 7D, 58.48% 30D), and validates economic value through portfolio simulations showing 17.01% net annual return and 1.51 Sharpe ratio. The approach is practical for event-driven trading strategies, sentiment-aware signal generation, and market-neutral portfolio construction. The interpretability analysis reveals which tokens benefit from time series context, useful for feature engineering in trading systems.

Moderately high complexity. Requires building on Llama3-8B with custom modality-specific expert layers (separate LN, QKV, MLP for time series), implementing selective cross-modal attention in deeper layers, designing the SALMON pretraining objective with STW mechanism, time series discretization with quantile binning, and LoRA fine-tuning. The architecture involves multiple interacting components (modality-specific experts, selective attention, cross-modal pretraining, salient token weighting). However, the code is released and the approach builds on standard transformer components.

Code is released at https://github.com/rosskoval/mlm_text_ts. Uses publicly available FNSPID dataset and HuggingFace pretrained checkpoints. Training details, hyperparameters, and hardware specifications (single NVIDIA H100 GPU) are provided. Statistical significance tests (DeLong's test) and multiple random seeds are reported. However, some baseline implementations require tuning choices that may affect exact reproduction.