Probability of Backtest Overfitting
“The Probability of Backtest Overfitting” (Bailey, Borwein, López de Prado & Zhu, 2015, Journal of Computational Finance) defines the PBO — the conditional probability that the strategy configuration that is optimal in-sample underperforms the median configuration out-of-sample — and estimates it with Combinatorially Symmetric Cross-Validation (CSCV), a model-free nonparametric procedure that swaps all in-sample/out-of-sample partitions of a backtested-returns matrix. The paper argues that a single hold-out is unreliable for investment backtests because it ignores the number of trials, is high-variance on short financial samples, and is contaminated when data is public. It is a research target realised in this vault as the practical complement to Out-of-Sample Backtesting for diagnosing Overfitting in Quantitative Trading.
Connections
- Overfitting in Quantitative Trading — defines, source: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253
- Pseudo-Mathematics and Financial Charlatanism — relates, source: https://www.ams.org/notices/201405/rnoti-p458.pdf
- Out-of-Sample Backtesting — supports, source: https://www.davidhbailey.com/dhbpapers/backtest-prob.pdf
- Marcos López de Prado — proposes_model, source: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2326253