State-Count Selection

State-count selection is the problem of choosing how many hidden regimes a Hidden Markov Model Regime Detection — or any Markov Regime-Switching Model — should have. It is the hidden-state analogue of State Definition Arbitrariness (which concerns the observed state buckets of a discrete Markov chain), and it sits under the same general State-Space Design failure mode. The number of states is a hyper-parameter, and the failure mode is that there is no robust, objective answer — different selection principles disagree, the choice is unstable, and, crucially, more states do not reliably help.

The principles pull in different directions. Statistical information criteria (AIC, BIC, likelihood-ratio tests) often favour three or four states because extra states always improve in-sample fit. Interpretability and estimation stability favour two states — a “calm” and a “turbulent” regime — which is why two-state HMMs dominate trading practice. A three-state model frequently produces a spurious extra regime that simply absorbs outliers rather than identifying a distinct economic state. Hamilton’s position in the founding regime-switching literature is unambiguous: parsimony is the remedy, because each added state must be estimated from observations that fall inside it, and a richly parameterised regime model becomes overfitted and misspecified — directly linking state-count selection to HMM Parameter Instability.

State-Count Selection [part-of] State-Space Design State-Count Selection [relates] State Definition Arbitrariness State-Count Selection [causes] HMM Parameter Instability

The decisive empirical evidence in this vault is Baitinger & Hoch 2024. They ran a controlled contest of HMM versus Hidden Semi-Markov Model on S&P 500 returns and explicitly varied the number of hidden states. Their finding: increasing the number of hidden states does not necessarily improve investment-strategy performance, and the richer HSMM’s advantage over the plain HMM — visible in-sample — “largely disappears in out-of-sample applications.” This is the textbook signature of Overfitting in Quantitative Trading: the extra states buy in-sample fit that is spent on noise and does not generalise. It generalises Dacco and Satchell 1999’s message — knowing the true model does not help if you misclassify — to: fitting a more flexible model does not help, because the flexibility is spent on in-sample noise.

Baitinger & Hoch 2024 [supports] State-Count Selection State-Count Selection [causes] Overfitting in Quantitative Trading

State-count selection therefore matters as a risk for two distinct reasons. As a data-snooping channel: if the state count is chosen by trying several and reporting the best, it inflates the effective number of strategies tested and the reported Sharpe ratio should be deflated accordingly (Data-Snooping Bias). As an overfitting trap: the instinct to equate a more elaborate state space — more regimes, semi-Markov durations — with a better trading model is, on the vault’s evidence, mistaken; the added structure is an in-sample artefact unless it demonstrably survives genuine out-of-sample testing. The verdict for this vault is confirmed: state-count selection is a real, unhedged degree of freedom in every regime-switching backtest, and the empirical record argues for parsimony — two states, validated out-of-sample — rather than sophistication as the path to anything tradeable.

State-Count Selection [causes] Data-Snooping Bias

Connections

• State-Space Design — part-of, hidden-state instance of the general failure mode, source: https://arxiv.org/html/2410.14504v2
• State Definition Arbitrariness — relates, observed-state analogue for discrete Markov chains, source: https://www.sciencedirect.com/science/article/abs/pii/S0957417419303033
• Hidden Markov Model Regime Detection — suffers_overfitting_risk, source: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4796238
• Baitinger & Hoch 2024 — supports, more hidden states do not improve out-of-sample investment performance, source: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4796238
• HMM Parameter Instability — causes, more states means thinner per-state data, source: https://arxiv.org/html/2402.05272v2
• Overfitting in Quantitative Trading — suffers_overfitting_risk, extra states buy in-sample fit that does not generalise, source: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4796238
• Data-Snooping Bias — suffers_overfitting_risk, choosing the best state count inflates the effective strategy count, source: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4796238
• Dacco and Satchell 1999 — relates, generalises “true model does not help” to “richer model does not help”, source: https://www.nber.org/system/files/working_papers/w21863/w21863.pdf

Sources

• Baitinger & Hoch (2024), Simplicity versus Complexity: HMM vs HSMM for Regime-Based Asset Allocation, SSRN 4796238
• Shu, Yu & Mulvey (2024), Downside Risk Reduction Using Regime-Switching Signals, Journal of Asset Management (arXiv:2402.05272)
• Hamilton (2016), Macroeconomic Regimes and Regime Shifts, NBER WP 21863