Level 3 AI in trading · Part 7/8
Backtest Overfitting: The Number of Trials Counts
This is the most important article of this level. Every method so far fixes an error in construction. This one deals with an error in evaluation, and it is the most expensive one.
The core problem
Suppose you test fifty strategies and not one of them has a real edge. Their Sharpe ratios scatter randomly around zero. The best of the fifty will still be clearly positive – not because of skill, but because the maximum of fifty draws is expected to sit far to the right.
The expected maximum grows with the square root of the logarithm of the number of trials. Fifty worthless attempts typically produce a best value of around 2.8 – a number that passes as excellent in any presentation.
Hence the central sentence of this chapter: a Sharpe ratio without a stated number of trials is not information.
Why the number of trials is almost always understated
Most people would deny having tested fifty variants. In fact, many more decisions count than you consciously register:
- every changed indicator period,
- every adjusted entry or exit rule,
- every switched period, every excluded instrument,
- every idea dropped after one look at the equity curve.
That last point is decisive. Abandoned attempts count too. Throwing out a variant after five seconds of looking at a curve is a completed test whose result you used.
Deflated Sharpe ratio
The antidote is a correction that explicitly accounts for the number of trials. The deflated Sharpe ratio asks one question: how likely is the observed Sharpe ratio to appear purely by chance, given that many attempts?
The calculation takes the observed Sharpe ratio, the length of the series, the number of trials, and the skewness and kurtosis of the return distribution. Those last two are not decoration. Strategies with many small gains and rare large losses – selling options, for example – produce Sharpe ratios that the classic formula systematically overstates.
The output is not a new marketing metric but a probability: how confident can you be that there is anything here at all?
Probability of backtest overfitting
A second approach that complements the first. You split the data repeatedly into training and test halves in different combinations. For each split, you take the variant that performed best in training and check how it performs on the test half.
If the best in-sample variant regularly lands below the median out of sample, the selection is overfitted. The share of such cases is the probability of backtest overfitting (PBO). As it approaches 50 percent, the procedure is selecting at random.
The practical value is that this does not evaluate a single strategy but the selection process. That is where the error lives.
What this means in practice
- Log your trials. Start counting before the first test and record every variant, including the ones you discard.
- Hypothesis before data. If you first argue why a relationship should exist and then test it, you need few trials. Searching needs many – and you have to pay for them.
- Be skeptical of best values. The one shining variant from a long series is, by construction, the most overfitted one.
- Report ranges, not points. A distribution is more honest than a single number – the same logic as in the Monte Carlo tool .
The uncomfortable conclusion
Most published backtests – in books, on social media, in product brochures – state no number of trials. By everything in this article, their metrics cannot be interpreted. That is not an accusation against individual authors but a structural feature: anyone who presents a result has almost always selected it from several.
Next
To close: what models can say about their own features, and how a probability becomes a position size.