{"format":"alysly-page/1","url":"https://alysly.com/learn/backtest-overfitting/","json":"https://alysly.com/json/learn/backtest-overfitting/index.json","lang":"en-US","title":"Backtest Overfitting: Deflated Sharpe Ratio and PBO","description":"Why the best of many backtests almost always looks good, how the deflated Sharpe ratio corrects for it, and what the probability of overfitting measures.","published":"2026-08-02T15:54:51+02:00","modified":"2026-11-04T09:15:31+01:00","alternates":[{"lang":"de-DE","url":"https://alysly.com/de/lernen/backtest-overfitting/","json":"https://alysly.com/json/de/lernen/backtest-overfitting/index.json"}],"about":"Machine-readable version of the page at url. It contains the text, tables and key figures published there, but not the interactive charts and tools.","breadcrumb":[{"name":"Investing & Trading Basics","url":"https://alysly.com/learn/"},{"name":"Level 3","url":"https://alysly.com/learn/#level-3"},{"name":"Backtest Overfitting: The Number of Trials Counts","url":"https://alysly.com/learn/backtest-overfitting/"}],"content":[{"type":"paragraph","text":"Level 3 AI in Trading · Part 8/9"},{"type":"heading","level":1,"text":"Backtest Overfitting: The Number of Trials Counts"},{"type":"paragraph","text":"By Alexander Bock, founder and data analyst","links":[{"text":"Alexander Bock","href":"https://alysly.com/about/"}]},{"type":"paragraph","text":"This is the most important article of this level. Every method so far fixes an error in construction. This one deals with an error in evaluation, and it is the most expensive one."},{"type":"heading","level":2,"id":"the-core-problem","text":"The Core Problem"},{"type":"paragraph","text":"Suppose you test fifty strategies and not one of them has a real edge. Their Sharpe ratios scatter randomly around zero. The best of the fifty will still be clearly positive. The reason is not skill. The maximum of fifty draws is expected to sit far to the right."},{"type":"image","src":"https://alysly.com/img/ta/best-of-n.d549c255b8b89377fc4d33d75bb4266cc33b9d3e591528e408e7f2f95f7e14bf.webp","alt":"Distribution of Sharpe ratios under pure chance versus the distribution of the best of 50 trials"},{"type":"paragraph","text":"(1) This is how the Sharpe ratio of ONE strategy with no edge is distributed — mean zero. (2) This is how the BEST of 50 equally worthless trials is distributed: its expectation is around 2.8. (3) A backtest with Sharpe 2.2 looks great against the left distribution and below average against the right one. That is why the number of variants tried is part of the result — hiding it means reporting a maximum and calling it skill."},{"type":"paragraph","text":"The expected maximum grows with the square root of the logarithm of the number of trials. Fifty worthless attempts typically produce a best value of around 2.8. That number passes as excellent in any presentation."},{"type":"paragraph","text":"Hence the key point of this chapter: a Sharpe ratio without a stated number of trials tells you nothing."},{"type":"heading","level":2,"id":"why-the-number-of-trials-is-almost-always-understated","text":"Why the Number of Trials Is Almost Always Understated"},{"type":"paragraph","text":"Most people would deny having tested fifty variants. In fact, many more decisions count than you consciously register:"},{"type":"list","ordered":false,"items":["every changed indicator period,","every adjusted entry or exit rule,","every switched period, every excluded instrument,","every idea dropped after one look at the equity curve."]},{"type":"paragraph","text":"That last point is decisive. Abandoned attempts count too. Throwing out a variant after five seconds of looking at a curve is a completed test whose result you used."},{"type":"heading","level":2,"id":"deflated-sharpe-ratio","text":"Deflated Sharpe Ratio"},{"type":"paragraph","text":"The antidote is a correction that explicitly accounts for the number of trials. The deflated Sharpe ratio asks one question: how likely is the observed Sharpe ratio to appear purely by chance, given the many attempts?"},{"type":"paragraph","text":"The calculation takes the observed Sharpe ratio, the length of the series, the number of trials, and the skewness and kurtosis of the return distribution. Those last two are not decoration. Strategies with many small gains and rare large losses, such as selling options, produce Sharpe ratios that the classic formula systematically overstates."},{"type":"paragraph","text":"The output is a probability: how confident can you be that there is anything here at all?"},{"type":"heading","level":2,"id":"probability-of-backtest-overfitting","text":"Probability of Backtest Overfitting"},{"type":"paragraph","text":"A second approach that complements the first. You split the data repeatedly into training and test halves in different combinations. For each split, you take the variant that performed best in training and check how it performs on the test half."},{"type":"paragraph","text":"If the best in-sample variant regularly lands below the median out of sample, the selection is overfitted. The share of such cases is the probability of backtest overfitting (PBO). As it approaches 50 percent, the procedure is selecting at random."},{"type":"paragraph","text":"The practical value is that this evaluates the selection process rather than a single strategy. That is where the error lives."},{"type":"heading","level":2,"id":"what-this-means-in-practice","text":"What This Means in Practice"},{"type":"list","ordered":false,"items":["Logged trials. Counting starts before the first test, and every variant is recorded, including discarded ones.","Hypothesis before data. A relationship that is argued for before it is tested needs few trials. Searching needs many, and those trials have a price.","Skepticism toward best values. The one shining variant from a long series is, by construction, the most overfitted one.",{"text":"Ranges instead of points. A distribution is more honest than a single number. The same logic applies in the Monte Carlo tool.","links":[{"text":"Monte Carlo tool","href":"https://alysly.com/tools/monte-carlo/"}]}]},{"type":"heading","level":2,"id":"the-uncomfortable-conclusion","text":"The Uncomfortable Conclusion"},{"type":"paragraph","text":"Most published backtests in books, on social media and in product brochures state no number of trials. By the logic of this article, their metrics cannot be interpreted. The problem is structural: anyone who presents a result has almost always selected it from several."},{"type":"heading","level":2,"id":"next","text":"Next"},{"type":"paragraph","text":"To close: what models can say about their own features, and how a probability becomes a position size."},{"type":"list","ordered":false,"items":[{"text":"Previous Cross-Validation Without Leakage","href":"https://alysly.com/learn/cross-validation/"},{"text":"Next Feature Importance and Bet Sizing","href":"https://alysly.com/learn/feature-importance/"}]},{"type":"heading","level":2,"text":"Indices"},{"type":"table","caption":"Indices, performance over the selected period","columns":["Index","1M"],"rows":[[{"text":"Dow Jones","href":"https://alysly.com/markets/indices/dow-jones/"},{"text":"+4.1%","data":{"m1":"+4.1%","m3":"-1.8%","m6":"+6.8%","ytd":"+10.9%","max":"+34.1%"}}],[{"text":"S&P 500","href":"https://alysly.com/markets/indices/sp500/"},{"text":"-1.0%","data":{"m1":"-1.0%","m3":"-0.4%","m6":"+4.5%","ytd":"+12.4%","max":"+37.8%"}}],[{"text":"MSCI World","href":"https://alysly.com/markets/indices/msci-world/"},{"text":"-0.3%","data":{"m1":"-0.3%","m3":"-0.6%","m6":"+3.6%","ytd":"+11.9%","max":"+37.4%"}}],[{"text":"MSCI ACWI","href":"https://alysly.com/markets/indices/msci-acwi/"},{"text":"-1.6%","data":{"m1":"-1.6%","m3":"-1.3%","m6":"+2.0%","ytd":"+12.0%","max":"+43.9%"}}]]},{"type":"heading","level":2,"text":"ETFs"},{"type":"table","caption":"ETFs, performance over the selected period","columns":["Fund","1M"],"rows":[[{"text":"SPDR S&P 500 ETF Trust SPY","links":[{"text":"SPDR S&P 500 ETF Trust","href":"https://alysly.com/markets/etfs/us/spy/"}]},{"text":"-1.0%","data":{"m1":"-1.0%","m3":"-0.4%","m6":"+4.5%","ytd":"+12.4%","max":"+37.8%"}}],[{"text":"Vanguard S&P 500 ETF VOO","links":[{"text":"Vanguard S&P 500 ETF","href":"https://alysly.com/markets/etfs/us/voo/"}]},{"text":"-1.1%","data":{"m1":"-1.1%","m3":"-0.4%","m6":"+4.5%","ytd":"+12.4%","max":"+36.2%"}}],[{"text":"iShares Core MSCI World UCITS ETF USD (Acc) EUNL","links":[{"text":"iShares Core MSCI World UCITS ETF USD (Acc)","href":"https://alysly.com/markets/etfs/ishares-core-msci-world-ucits-etf-usd-acc/"}]},{"text":"-2.0%","data":{"m1":"-2.0%","m3":"-2.3%","m6":"+4.3%","ytd":"+10.6%","max":"+487.5%"}}],[{"text":"Xtrackers MSCI Emerging Markets UCITS ETF 1c XMME","links":[{"text":"Xtrackers MSCI Emerging Markets UCITS ETF 1c","href":"https://alysly.com/markets/etfs/xtrackers-msci-emerging-markets-ucits-etf-1c/"}]},{"text":"-10.9%","data":{"m1":"-10.9%","m3":"-5.8%","m6":"-5.0%","ytd":"+12.1%","max":"+89.8%"}}]]},{"type":"heading","level":2,"text":"Stocks"},{"type":"table","caption":"Stocks, performance over the selected period","columns":["Stock","1M"],"rows":[[{"text":"NVIDIA Corporation NVDA","links":[{"text":"NVIDIA Corporation","href":"https://alysly.com/markets/stocks/nvidia-corporation/"},{"text":"United States","href":"https://alysly.com/markets/stocks/browse/?country=United%20States"}]},{"text":"-16.2%","data":{"m1":"-16.2%","m3":"-8.7%","m6":"-3.7%","ytd":"+7.3%","max":"+57.1%"}}],[{"text":"Apple Inc. AAPL","links":[{"text":"Apple Inc.","href":"https://alysly.com/markets/stocks/apple-inc/"},{"text":"United States","href":"https://alysly.com/markets/stocks/browse/?country=United%20States"}]},{"text":"+5.0%","data":{"m1":"+5.0%","m3":"+12.4%","m6":"+21.6%","ytd":"+28.6%","max":"+53.6%"}}],[{"text":"Tesla Inc. TSLA","links":[{"text":"Tesla Inc.","href":"https://alysly.com/markets/stocks/tesla-inc/"},{"text":"United States","href":"https://alysly.com/markets/stocks/browse/?country=United%20States"}]},{"text":"+1.7%","data":{"m1":"+1.7%","m3":"+19.8%","m6":"-3.4%","ytd":"-14.3%","max":"+59.8%"}}]]},{"type":"paragraph","text":"Data note 1","href":"#dn-1"},{"type":"heading","level":2,"text":"More Studies"},{"type":"list","ordered":false,"items":[{"text":"Jul 21, 2026 Collectibles ×5.4 for the same cards, matched-model index, Mar 2022 — Aug 2026 Is the Pokémon Card Market in a Bubble? 4,159 Sales Analyzed Prices for PSA-graded cards multiplied. Measured on the same cards they rose 438%, while the raw median shows 786% because pricier cards came up for sale later.","href":"https://alysly.com/analysis/pokemon-card-market-bubble/"},{"text":"May 19, 2026 Crypto 45.6% Bitcoin’s annualized volatility, Jul 2024 — Sep 2026 Bitcoin in Investment Portfolios: The Real Numbers Bitcoin is calmer than it was and still about three times as volatile as the stock market. It moves with stocks, not with gold.","href":"https://alysly.com/analysis/bitcoin-investment-portfolios/"},{"text":"Mar 17, 2026 Wealth €1.2m needed for €3,000 a month indefinitely at a 3% real return Living Off Your Returns: What It Really Takes The capital you need is arithmetic. The return you assume is a guess, and it can triple the answer.","href":"https://alysly.com/analysis/living-off-returns/"}]},{"type":"paragraph","text":"View all studies","href":"https://alysly.com/analysis/#studies"},{"type":"heading","level":2,"text":"Learn"},{"type":"list","ordered":false,"items":[{"text":"Feature Importance and Bet Sizing Which features actually drive the model — and how that becomes a position size.","href":"https://alysly.com/learn/feature-importance/"},{"text":"What Are ETFs? The Basics Explained An ETF bundles many stocks into one fund that trades on an exchange like a single stock. How ETFs work, what they cost and how to see what they hold.","href":"https://alysly.com/learn/what-are-etfs/"}]},{"type":"paragraph","text":"Course overview","href":"https://alysly.com/learn/"},{"type":"paragraph","text":"Updated Nov 4, 2026 · Content and figures checked Oct 19, 2026"},{"type":"heading","level":2,"id":"note-legend-h","text":"Data Notes"},{"type":"paragraph","text":"All calculations run automatically and are not investment advice."},{"type":"list","ordered":true,"items":[{"id":"dn-1","text":"Returns computed automatically from the price series, excluding dividends, taxes and costs."}]},{"type":"paragraph","text":"Data and methodology: Alysly · Responsible for content: Alexander Bock","links":[{"text":"Alysly","href":"https://alysly.com/methodology/"},{"text":"Alexander Bock","href":"https://alysly.com/about/"}]},{"type":"paragraph","text":"View all data notes","href":"https://alysly.com/notes/"}],"structured_data":[{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://alysly.com/#org","name":"Alysly","url":"https://alysly.com/","logo":{"@type":"ImageObject","url":"https://alysly.com/img/icon-512.png","width":512,"height":512},"description":"Data-driven insights, transparent analysis and powerful tools for long-term investors.","email":"mail@alysly.com","contactPoint":{"@type":"ContactPoint","contactType":"customer support","email":"mail@alysly.com","availableLanguage":["de","en"]},"address":{"@type":"PostalAddress","addressLocality":"Haltern am See","postalCode":"45721","addressCountry":"DE"},"founder":{"@id":"https://alysly.com/#founder"},"sameAs":["https://www.linkedin.com/company/alyslycom/","https://www.youtube.com/@alyslycom","https://x.com/alyslycom"],"publishingPrinciples":"https://alysly.com/methodology/"},{"@type":"Person","@id":"https://alysly.com/#founder","name":"Alexander Bock","jobTitle":"Founder and data analyst","url":"https://alysly.com/about/","sameAs":["https://www.linkedin.com/in/abxk/"]},{"@type":"WebSite","@id":"https://alysly.com/#site","url":"https://alysly.com/","name":"Alysly","alternateName":"Alysly Investment Analysis","inLanguage":"en-US","publisher":{"@id":"https://alysly.com/#org"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://alysly.com/markets/stocks/browse/?q={search_term_string}"},"query-input":"required name=search_term_string"}},{"@type":"WebPage","@id":"https://alysly.com/learn/backtest-overfitting/#webpage","url":"https://alysly.com/learn/backtest-overfitting/","name":"Backtest Overfitting: The Number of Trials Counts – Alysly","description":"Why the best of many backtests almost always looks good, how the deflated Sharpe ratio corrects for it, and what the probability of overfitting measures.","inLanguage":"en-US","isPartOf":{"@id":"https://alysly.com/#site"},"dateModified":"2026-11-04T09:15:31+01:00","publisher":{"@id":"https://alysly.com/#org"},"mainEntity":{"@id":"https://alysly.com/learn/backtest-overfitting/#entity"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://alysly.com/img/ta/best-of-n-og.3b1d5bfb4c0d964aaa86707d6268cabe50ef8b79b6630e5a7deca4c76074b646.png","width":1200,"height":630}}]},{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Investing & Trading Basics","item":"https://alysly.com/learn/"},{"@type":"ListItem","position":2,"name":"Backtest Overfitting: The Number of Trials Counts"}]},{"@context":"https://schema.org","@type":"Article","@id":"https://alysly.com/learn/backtest-overfitting/#entity","headline":"Backtest Overfitting: The Number of Trials Counts","description":"Why the best of many backtests almost always looks good, how the deflated Sharpe ratio corrects for it, and what the probability of overfitting measures.","inLanguage":"en-US","mainEntityOfPage":"https://alysly.com/learn/backtest-overfitting/","image":[{"@type":"ImageObject","height":630,"url":"https://alysly.com/img/ta/best-of-n-og.3b1d5bfb4c0d964aaa86707d6268cabe50ef8b79b6630e5a7deca4c76074b646.png","width":1200},{"@type":"ImageObject","height":500,"url":"https://alysly.com/img/ta/best-of-n.d549c255b8b89377fc4d33d75bb4266cc33b9d3e591528e408e7f2f95f7e14bf.webp","width":1280}],"isPartOf":{"@type":"CreativeWorkSeries","name":"Investing & Trading Basics","url":"https://alysly.com/learn/"},"educationalLevel":"Level 3","author":{"@id":"https://alysly.com/#founder"},"publisher":{"@id":"https://alysly.com/#org"},"datePublished":"2026-08-02T15:54:51+02:00","dateModified":"2026-11-04T09:15:31+01:00"}],"notice":"© 2026 Alysly.com. Not investment advice. All data without guarantee. Financial information is analyzed and presented partly with AI assistance."}