Improving S&P 500 Volatility Forecasting through Regime-Switching Methods

Battle Count: 81

Highly relevant to quantitative trading as it directly addresses volatility forecasting, which is fundamental to options pricing, risk management, position sizing, and trading strategy development. The regime-switching approach is particularly valuable for adapting trading strategies to different market conditions. The dual-recursive RV-VIX framework captures feedback effects between implied and realized volatility, relevant for volatility trading strategies. The evaluation across crisis and normal periods demonstrates practical robustness. However, the marginal improvement over feature-engineered HAR and the complexity of implementation may limit direct deployment without further engineering.

The paper implements three distinct regime-switching pipelines, each requiring multiple stages: data preprocessing (z-score normalization, RV computation from 5-min returns), statistical testing (Mood's Median Test), clustering (spectral clustering with Wasserstein distances, Bayesian GMM, PCA), model fitting (regime-specific HAR via WLS), classification (XGBoost for regime probability prediction), and rolling window retraining. The dual-recursive forecasting adds further complexity with joint RV-VIX modeling. Multiple hyperparameters need tuning across methods. The use of optimal transport (POT library), HMM (hmmlearn), and gradient boosting adds to the engineering burden.

The paper provides detailed mathematical formulations, feature descriptions (Appendix A), XGBoost hyperparameter search spaces (Appendix B), and lists all Python libraries used (Appendix C). However, no code repository is provided, and specific hyperparameter values selected via cross-validation are not reported. The data source (Bloomberg Terminal) is proprietary and not freely accessible. The rolling window size (441 days), number of regimes, and other hyperparameters are described but exact tuning results are not fully disclosed.