HMM Regime Forecasting, Look-Ahead Bias, and the Role of KS Tests
Summary
The document raises two questions about reproducing a hidden Markov model study of factor investing across market regimes. First, it asks whether the reported regime labels and their return and volatility summaries use future observations to classify past dates. This highlights a key distinction in HMM inference: smoothed state probabilities use observations from the full sample, while filtered probabilities use only information available at the time. A backtest intended to reflect live decisions needs a causal estimate.
Second, the author questions how a Kolmogorov–Smirnov test and fitted observation distributions contribute to detecting a regime change, given that an HMM already has a transition matrix. The document presents the paper’s description of comparing new return and volatility observations with regime-specific distributions, but gives no answer or replication results. It therefore identifies issues to investigate rather than validating the proposed mechanism, and leaves unclear how the distribution fits, state probabilities, and transition forecasts are combined in the backtest.
Key ideas
- HMM state smoothing can use future observations, unlike filtering based only on information available at the time.
- A regime backtest must clarify whether it uses filtered or smoothed state probabilities.
- The paper describes fitting distributions to regime observations and comparing new data against them.
- The document questions how this procedure relates to the HMM transition matrix but provides no resolution.
Tags
Full text
# 66227 # Trying to recreate results from a research paper on HMM and Kolmogorov-Smirnov Test for forecasting regime switching on SP500 I am trying to recreate this research: Regime-Switching Factor Investing with Hidden Markov Models, by Matthew Wang, Yi-Hong Lin and Ilya Mikhelson https://www.mdpi.com/1911-8074/13/12/311/htm My first question is when they present the regime breakdown: ``` Regime Distribution Return Volatility 0 0.4237 0.0398 3.4739 1 0.4463 0.04635 0.9438 2 0.13 -0.066 13.63465 ``` These results look great, but don't they suffer from hindsight bias? They're giving the algorithm all of the data so it can look at future states to calculate probability of a past state. For example, say it correctly identifies Sept 2008 as state2 with low returns and high volatility. But to do this, the algorithm used information from Oct 2008 and onwards when determining Sept 2008 was state2? Or am I misunderstanding how this works? My second question is when they turn to backtesting: > "Once we have the trained HMM, we need to design a mechanism that serves as the aforementioned confidence interval for judging whether a regime switch has occurred. In using the GaussianHMM API from hmmlearn we operate under the assumption that the observation probability distributions for volatility and daily returns are normally distributed. However, in the mechanism we designed for regime detection, we analyzeg regime observations independently. Thus, as can be observed in Figure 5, we utilize the Kolmogorov-Smirnov test to fit the observations to one of many common distributions including normal, lognormal, pareto, gamma, beta and exponential. With new daily returns and volatility values, we are then able to use the probability density function (PDF) of the fitted distribution to compare the likelihood we are currently in each respective regime in relative terms." I don't understand how they are using a K-S test to use the output from the HMM model into a probability of the regime switching in the future. Doesn't the HMM transition matrix tell you this? Ie. knowing that we are likely in State 1, there is an X probability of staying in State 1 etc?
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.