Level 3 AI in trading · Part 7/8
Backtest Overfitting: the Number of Trials Counts
This is the most important article of this level. Every method so far fixes an error in construction. This one addresses an error in evaluation – and it is the most expensive.
The core problem
Suppose fifty strategies are tested and not one of them has a genuine edge. Their Sharpe ratios scatter randomly around zero. The best of the fifty will still be clearly positive – not through skill, but because the maximum of fifty draws is expected to sit far to the right.
The expected maximum grows with the square root of the logarithm of the number of trials. Fifty worthless attempts typically produce a best value around 2.8 – a number that passes as excellent in any presentation.
Hence the central sentence of this chapter: a Sharpe ratio without a stated number of trials is not information.
Why the trial count is almost always understated
Most practitioners would deny having tested fifty variants. In fact many more decisions count than are consciously registered:
- every changed indicator period,
- every adjusted entry or exit rule,
- every switched period, every excluded instrument,
- every idea dropped after one look at the equity curve.
That last point is decisive. Abandoned attempts count too. Discarding a variant after five seconds of looking at a curve is a completed test whose result you used.
Deflated Sharpe ratio
The antidote is a correction that explicitly accounts for the number of trials. The deflated Sharpe ratio asks: how likely is the observed Sharpe ratio to arise purely by chance given that many attempts?
The calculation takes in the observed Sharpe ratio, the length of the series, the number of trials, and the skewness and kurtosis of the return distribution. Those last two are not decoration – strategies with many small gains and rare large losses (selling options, for instance) produce Sharpe ratios that the classical formula systematically overstates.
The output is not a new marketing metric but a probability: how confident can you be that there is anything here at all?
Probability of backtest overfitting
A second, complementary approach. The data is split repeatedly into training and test halves in varying combinations. For each split, the variant that performed best in training is identified and its performance on the test half is checked.
If the in-sample best variant regularly lands below the median out of sample, the selection is overfitted. The share of such cases is the probability of backtest overfitting (PBO). As it approaches 50 per cent, the procedure is selecting at random.
The practical value is that this evaluates not a single strategy but the selection process. That is where the error lives.
What this means in practice
- Log your trials. Start a count before the first test and record every variant, including the discarded ones.
- Hypothesis before data. Arguing first why a relationship should exist, then testing, requires few trials. Searching requires many – and you have to pay for them.
- Be sceptical of best values. The one shining variant from a long series is by construction the most overfitted.
- Report ranges, not points. A distribution is more honest than a number – the same logic as the Monte Carlo tool .
The uncomfortable conclusion
Most published backtests – in books, on social media, in product brochures – state no trial count. By everything in this article, their metrics are therefore uninterpretable. That is not an accusation against individual authors but a structural feature: anyone presenting a result has almost always selected it from several.
Next
To close: what models can say about their own features – and how a probability becomes a position size.