SignalTrace

Markov Regime-Switching Model

John Montgomery7 min read

Markov Regime-Switching Model

The Markov regime-switching model is an econometric model in which the parameters of a time-series process — typically an autoregression or a volatility (GARCH-type) model — are not constant but switch between a small number of values according to an unobserved discrete state. That state follows a first-order Markov chain: the probability of moving to a given regime next period depends only on the current regime, through a fixed transition probability matrix p_ij. The lineage runs from switching-regression work by Quandt 1958 1972 and Goldfeld and Quandt 1973 to Hamilton 1989, whose paper on autoregressive models with Markov-switching parameters made the framework the standard tool for modelling business cycles and is the canonical reference for the whole family. The survey Macroeconomic Regimes and Regime Shifts by James D. Hamilton gives the modern formulation: conditional on the latent state the model is linear and as tractable as an ordinary AR model, but because the state is hidden, the observable process is nonlinear — the optimal forecast must average over the inferred probability of each regime.

Hamilton 1989 defines Markov Regime-Switching Model James D. Hamilton proposes_model Markov Regime-Switching Model Macroeconomic Regimes and Regime Shifts defines Markov Regime-Switching Model Markov Regime-Switching Model part-of Regime Classification

It is worth being precise about how this model differs from its two siblings in this vault. A Hidden Markov Model Regime Detection note covers what is, mathematically, the same object — econometricians use “Markov-switching model” and “hidden Markov model” almost interchangeably (Bulla et al. explicitly call HMM “a synonym”). The conventional distinction is one of emphasis and discipline: “regime-switching” denotes the econometrics tradition where the switching object is a parametric structural model (an AR or VAR or GARCH equation) of a macro or financial series, the regimes are economically interpreted (recession, bear market), and the goal is explanation and inference; “hidden Markov model” denotes the engineering/ML tradition emphasising the latent-state machinery itself and its decoding (Viterbi, forward-backward) for pattern recognition. The Markov Chain Trading Model sibling is genuinely different: it applies a Markov chain directly to observed discretised price moves (up/down/flat states) with no hidden layer and no switching structural equation — a much weaker construct that the efficient-market literature largely rejects for direct prediction. The regime-switching model’s strength is precisely the hidden structural layer the plain Markov-chain model lacks.

Markov Regime-Switching Model relates Hidden Markov Model Regime Detection Markov Regime-Switching Model contradicts Markov Chain Trading Model Markov Regime-Switching Model relates Quandt 1958 1972 Goldfeld and Quandt 1973 precedes Hamilton 1989

The model’s established uses are descriptive and explanatory econometrics, and here the evidence is strong. Hamilton’s two-state model of US GDP, estimated from output data alone, recovers expansion/recession dating that closely matches the NBER business-cycle chronology, and the one-quarter-ahead regime probabilities track NBER recessions well even out-of-sample. In finance the same machinery cleanly separates calm and turbulent volatility regimes, bull and bear markets, and has been used to study time-varying factor premia and correlations. Its established trading use is narrower and should be stated honestly: it is a Regime-Based Asset Allocation risk filter — detect a high-volatility/bear regime and de-risk into cash or bonds — not a directional return-forecasting engine. Bulla et al. 2010 (“Markov-switching Asset Allocation: Do Profitable Strategies Exist?”) is the most disciplined positive test: a two-state Gaussian model on daily US/German/Japanese equity indices over 40 years, run out-of-sample with transaction costs of 10 bp one-way explicitly charged. It answers its own title question “Yes” — but the magnitude must be graded carefully: out-of-sample annual excess return over buy-and-hold was only 18.5 bp for the S&P 500, up to 201.6 bp for the Nikkei; the real prize was a ~41% cut in volatility and large Sharpe-ratio improvement. The Nikkei strategy still lost money in absolute terms (-2.28%/yr), just less than the index (-4.30%). This is a credible risk-management result, not evidence of large tradeable alpha.

Bulla et al. 2010 tests_strategy Regime-Based Asset Allocation Bulla et al. 2010 reports_profitability Markov Regime-Switching Model Bulla et al. 2010 includes_costs Transaction Costs and Slippage Bulla et al. 2010 uses_dataset Major Equity Indices Daily Returns Regime-Based Asset Allocation detects_regime Markov Regime-Switching Model

That carefully-hedged positive result only exists because the study suppressed the model’s central failure mode, and the negative evidence is at least as important as the positive. Dacco and Satchell 1999 (“Why do regime-switching models forecast so badly?”) proves analytically that forecasts from the true regime-switching model can have higher mean squared error than a random walk: forming a forecast requires classifying the current regime, and even a small misclassification rate is enough to destroy any advantage from knowing the correct model. This is the gap between ex-post smoothed regime inference (sharp, uses the full sample, available only with hindsight) and real-time filtered inference (the only thing a trader actually has, and noisy). The jump-model study of Shu Yu and Mulvey 2024 quantifies the lag: regime detection has a median latency around 25 days; during the COVID-19 crash the online signal missed the rebound and did not improve returns — it only avoided a ~20% drawdown. Bulla et al. only stayed profitable by engineering turnover down (daily data, Viterbi paths, a 6-day median filter cutting switches 50-65%) and by prior work (Hess 2006, Ammann & Verhofen 2006) showing the edge vanishes after costs without such tricks. These turnover knobs are themselves free parameters, raising data-snooping and overfitting concerns.

Dacco and Satchell 1999 contradicts Markov Regime-Switching Model Real-Time Regime Identification Lag causes Parameter Instability and Estimation Noise Dacco and Satchell 1999 supports Real-Time Regime Identification Lag Shu Yu and Mulvey 2024 tests_strategy Statistical Jump Model

The remaining failure modes are structural and well-documented by Hamilton himself. Number of regimes: the likelihood-ratio test of “N vs N+1 regimes” does not have the usual chi-square distribution because nuisance parameters (the transition probabilities) are unidentified under the null — so practitioners cannot cleanly test how many regimes the data support, and adding regimes to improve fit invites overfitting. Few observations per regime: parameters of a regime can only be learned from data inside that regime; with ~11 postwar US recessions, a richly parameterised transition model is, in Hamilton’s words, “overfitted and misspecified.” Multiple local maxima: the EM likelihood must be restarted from many points or estimation converges to spurious solutions — a source of parameter instability. Hamilton’s own recommendation is parsimony: few regimes, few switching parameters (typically just the intercept and variance). The honest verdict for this vault: the Markov regime-switching model is confirmed as valuable descriptive/explanatory econometrics and as a real-time risk filter that reliably cuts volatility and drawdown; it is not substantiated as a source of large directional trading profit, and naive backtests that omit costs, use smoothed (hindsight) regimes, or stack regimes for fit will overstate its edge.

Markov Regime-Switching Model suffers_overfitting_risk Overfitting in Quantitative Trading James D. Hamilton opposes Overfitting in Quantitative Trading Parameter Instability and Estimation Noise contradicts Markov Regime-Switching Model Markov Regime-Switching Model supports Regime Classification

Connections

Sources

Graph View

Backlinks