PRAGMA: Revolut Foundation Model

Battle Count: 57

Moderately relevant. PRAGMA focuses on consumer banking event sequences (credit scoring, fraud, LTV) rather than market data or trading signals. However, the key-value-time tokenisation approach for heterogeneous structured event data, the masked modelling pre-training strategy, and the encoder-only architecture for discriminative tasks could inspire similar approaches for modeling trading event streams, order book data, or transaction-level market microstructure. The profile state + event history fusion architecture is conceptually transferable to portfolio-level modeling.

High complexity. Requires: (1) custom key-value-time tokenisation with type-specific encoding (percentile bucketing, BPE, categorical mapping), (2) three-branch encoder architecture (Profile State, Event, History) with RoPE positional encoding, (3) specialized data storage (LMDB + Parquet shards), (4) dynamic batching with sequence packing and varlen attention kernels, (5) multi-level truncation strategies, (6) MLM with three masking sources, (7) LoRA fine-tuning infrastructure, (8) 16-32 H100 GPUs for training. The engineering pipeline for handling 24B events across 111 countries is substantial.

Very low reproducibility. All data is commercially sensitive and proprietary to Revolut. Only relative improvements are reported, not absolute metrics. All examples shown are synthetic and not from real production data. No code or model weights are released. The pre-training corpus (26M users, 24B events, 207B tokens) is not available. Training infrastructure details (16-32 H100 GPUs) are provided but the data pipeline is proprietary.