SignalTrace

Data-Snooping Bias

John Montgomery5 min read

Data-Snooping Bias

Data-snooping bias (also data-mining bias or selection bias) is the inflation of apparent performance that arises when a dataset is used more than once for inference or model selection, and only the best-performing results are reported. Even when no strategy in the search has a genuine edge, the maximum backtest Sharpe ratio over a large grid grows simply by chance — so a “profitable” Markov strategy may be the survivor of an unreported search rather than a real signal. It is the cross-strategy, multiple-testing counterpart of Overfitting in Quantitative Trading: overfitting tunes one model to noise; data snooping selects the luckiest model from many.

The canonical fix is Halbert White’s “A Reality Check for Data Snooping” (Econometrica, 2000). White provided the first general bootstrap procedure for testing the null hypothesis that the best model encountered in a specification search has no predictive superiority over a benchmark — the test accounts for the dependence among the candidate models’ statistics, so a search can be conducted “with some degree of confidence that one will not mistake results that could have been generated by chance for genuinely good results.” Applying the Reality Check, Sullivan, Timmermann and White found that technical trading rules lost their predictive power for major U.S. indices after the mid-1980s. Peter Hansen’s “A Test for Superior Predictive Ability” (SPA, Journal of Business and Economic Statistics, 2005) sharpened this: White’s Reality Check is conservative because its null distribution is anchored at the least-favourable configuration, so the presence of many poor, irrelevant models drags down the test’s power and can mask a genuine winner. The studentised SPA test, and the stepwise extensions of Romano & Wolf (2005) and Hsu et al. (2010), are more powerful and identify which models are superior.

Markov-model studies are especially exposed because the analyst chooses state definitions, lookback windows, the discretisation of price moves into Markov-chain buckets, the HMM state count, the Markov Regime-Switching Model regime count, the MDP state-space, and the RL reward function — each a search dimension — and the literature as a whole is a vast uncontrolled search across thousands of papers on overlapping datasets (a handful of major indices, currency pairs and crypto histories). Harvey, Liu and Zhu’s “…and the Cross-Section of Expected Returns” (Review of Financial Studies, 2016) made this concrete for asset pricing: they census a “factor zoo” of several hundred published return predictors and argue that, given the collective data mining, the conventional single-test hurdle of a t-statistic above 2.0 is meaningless — a newly claimed factor should clear a t-ratio above roughly 3.0. The same logic applies to any new Markov strategy: it is one more draw from an enormous, mostly unreported search.

For practitioners, Campbell Harvey and Yan Liu translate the multiple-testing framework into a Sharpe ratio haircut (“Backtesting”, 2015; “Evaluating Trading Strategies”, Journal of Portfolio Management, 2014). The routine practice of halving a backtested Sharpe ratio is, they show, only a rule of thumb: the correct haircut depends on the number of tests and grows non-linearly with it, applying Bonferroni, Holm and Benjamini-Hochberg-Yekutieli corrections adapted to correlated tests. The haircut Sharpe ratio is the value that would have resulted from a single test. Bailey and López de Prado’s Deflated Sharpe Ratio is the closely related tool that deflates a Sharpe ratio for selection bias under multiple testing together with non-normality of returns. McLean and Pontiff (2016) supply the empirical corroboration: academic factors lose roughly half their return after publication, exactly the decay data snooping predicts.

This note is central to the vault because it is the reason the vault privileges out-of-sample evidence, independent replication and explicit multiple-testing corrections, and downgrades any Markov backtest that does not disclose how many configurations or how much prior literature its result was selected from. No Markov-model paper reviewed so far reports a White Reality Check, an SPA test, or a Sharpe-ratio haircut — which is itself a finding about the reliability of the field.

Data-Snooping Bias [causes] Overfitting in Quantitative Trading Data-Snooping Bias [contradicts] Out-of-Sample Backtesting Halbert White [defines] Data-Snooping Bias Deflated Sharpe Ratio [opposes] Data-Snooping Bias And the Cross-Section of Expected Returns [supports] Data-Snooping Bias

Connections

Sources

Graph View

Backlinks