Explainable ML for Stock Market Volatility
What this project solves.
Academic research project applying XGBoost, SHAP explainability, Ruptures change-point detection, and Scikit-Learn to model dynamic stock market volatility across multiple countries. Focused on interpretable AI and financial forecasting.
Traditional financial volatility models treat markets as stationary systems with fixed dynamics. In reality, market regimes shift dramatically due to economic crises, policy changes, and geopolitical events — making static models unreliable for real-world forecasting.
The research applies Ruptures change-point detection to automatically segment market data into distinct regimes, then trains XGBoost models per regime. SHAP values provide full explainability — making it possible to understand exactly which features drive volatility predictions at each point in time.
The framework outperforms static baseline models and, crucially, provides interpretable explanations that financial analysts can audit and trust — addressing the 'black box' problem that limits AI adoption in regulated financial environments.
Under the hood
Features
- Automated market regime detection using Ruptures change-point algorithms
- Per-regime XGBoost volatility forecasting models
- Full SHAP explainability with feature importance visualisations
- Multi-country dataset analysis (US, UK, India, Sri Lanka)
- Comparative benchmark against GARCH and standard ML baselines
- Interactive visualisation of regime transitions and SHAP waterfall plots
- Reproducible experiment pipeline with configuration files
Challenges
- Financial time-series data has complex autocorrelations that violate standard ML train/test split assumptions
- Choosing the right number of change-points without overfitting to noise in the data
- Ensuring SHAP values remain consistent and interpretable across different market regimes
- Handling missing data and different trading calendars across multi-country datasets
Solutions
- Implemented walk-forward cross-validation with embargo periods to respect temporal ordering
- Used Bayesian Information Criterion (BIC) to objectively select the optimal number of change-points
- Computed SHAP values independently per regime model so explanations reflect each market context
- Built a robust data preprocessing pipeline with forward-fill, calendar alignment, and outlier detection
Lessons Learned
- Explainability is not optional in finance — it is a prerequisite for model adoption by practitioners
- Change-point detection fundamentally improves forecasting by acknowledging that markets evolve
- Walk-forward validation reveals true out-of-sample performance that k-fold cross-validation masks in time series
- Visualising SHAP values is as important as computing them — communication of results drives impact
