Level 3 AI in Trading · Part 8/9
Backtest Overfitting: The Number of Trials Counts
This is the most important article of this level. Every method so far fixes an error in construction. This one deals with an error in evaluation, and it is the most expensive one.
The Core Problem
Suppose you test fifty strategies and not one of them has a real edge. Their Sharpe ratios scatter randomly around zero. The best of the fifty will still be clearly positive. The reason is not skill. The maximum of fifty draws is expected to sit far to the right.
The expected maximum grows with the square root of the logarithm of the number of trials. Fifty worthless attempts typically produce a best value of around 2.8. That number passes as excellent in any presentation.
Hence the key point of this chapter: a Sharpe ratio without a stated number of trials tells you nothing.
Why the Number of Trials Is Almost Always Understated
Most people would deny having tested fifty variants. In fact, many more decisions count than you consciously register:
- every changed indicator period,
- every adjusted entry or exit rule,
- every switched period, every excluded instrument,
- every idea dropped after one look at the equity curve.
That last point is decisive. Abandoned attempts count too. Throwing out a variant after five seconds of looking at a curve is a completed test whose result you used.
Deflated Sharpe Ratio
The antidote is a correction that explicitly accounts for the number of trials. The deflated Sharpe ratio asks one question: how likely is the observed Sharpe ratio to appear purely by chance, given the many attempts?
The calculation takes the observed Sharpe ratio, the length of the series, the number of trials, and the skewness and kurtosis of the return distribution. Those last two are not decoration. Strategies with many small gains and rare large losses, such as selling options, produce Sharpe ratios that the classic formula systematically overstates.
The output is a probability: how confident can you be that there is anything here at all?
Probability of Backtest Overfitting
A second approach that complements the first. You split the data repeatedly into training and test halves in different combinations. For each split, you take the variant that performed best in training and check how it performs on the test half.
If the best in-sample variant regularly lands below the median out of sample, the selection is overfitted. The share of such cases is the probability of backtest overfitting (PBO). As it approaches 50 percent, the procedure is selecting at random.
The practical value is that this evaluates the selection process rather than a single strategy. That is where the error lives.
What This Means in Practice
- Logged trials. Counting starts before the first test, and every variant is recorded, including discarded ones.
- Hypothesis before data. A relationship that is argued for before it is tested needs few trials. Searching needs many, and those trials have a price.
- Skepticism toward best values. The one shining variant from a long series is, by construction, the most overfitted one.
- Ranges instead of points. A distribution is more honest than a single number. The same logic applies in the Monte Carlo tool.
The Uncomfortable Conclusion
Most published backtests in books, on social media and in product brochures state no number of trials. By the logic of this article, their metrics cannot be interpreted. The problem is structural: anyone who presents a result has almost always selected it from several.
Next
To close: what models can say about their own features, and how a probability becomes a position size.