{"format":"alysly-page/1","url":"https://alysly.com/learn/testing-a-pattern/","json":"https://alysly.com/json/learn/testing-a-pattern/index.json","lang":"en-US","title":"Testing Chart Patterns: Base Rates, Samples and Backtest Traps","description":"Turning a pattern into a testable rule: base rate, sample size, look-ahead bias, survivorship bias and the question of how much a backtest really shows.","published":"2026-08-02T15:41:33+02:00","modified":"2026-11-01T18:26:04+01:00","alternates":[{"lang":"de-DE","url":"https://alysly.com/de/lernen/muster-pruefen/","json":"https://alysly.com/json/de/lernen/muster-pruefen/index.json"}],"about":"Machine-readable version of the page at url. It contains the text, tables and key figures published there, but not the interactive charts and tools.","breadcrumb":[{"name":"Investing & Trading Basics","url":"https://alysly.com/learn/"},{"name":"Level 2","url":"https://alysly.com/learn/#level-2"},{"name":"Testing a Pattern","url":"https://alysly.com/learn/testing-a-pattern/"}],"content":[{"type":"paragraph","text":"Level 2 Patterns & Indicators · Part 9/9"},{"type":"heading","level":1,"text":"Testing a Pattern"},{"type":"paragraph","text":"By Alexander Bock, founder and data analyst","links":[{"text":"Alexander Bock","href":"https://alysly.com/about/"}]},{"type":"paragraph","text":"Up to this point the course has been descriptive. This article is the transition: it turns an observation into a statement that can be wrong."},{"type":"heading","level":2,"id":"step-1-make-the-rule-unambiguous","text":"Step 1: Make the Rule Unambiguous"},{"type":"paragraph","text":"A pattern is only testable when a program can find it without judgment. “A hammer after a downtrend” is not enough. A testable version would be:"},{"type":"paragraph","text":"Lower wick at least twice the body, upper wick at most a quarter of the body, and the close below the 20-period average."},{"type":"paragraph","text":"Every number in that definition is a decision. Those decisions become the problem later – but without them there is nothing to measure."},{"type":"heading","level":2,"id":"step-2-define-the-outcome","text":"Step 2: Define the Outcome"},{"type":"paragraph","text":"What exactly are you measuring? “The price rises afterwards” is not a definition. You need three things: a horizon, a threshold and a comparison. For example: is the close 10 periods later at least 1 ATR higher?"},{"type":"paragraph","text":"A worse version is surprisingly common: the price is only checked after a fixed horizon. A path that falls far below any sensible stop in between and recovers at the end still counts as a success. In practice, you would have been stopped out. Level 3 shows how to do this better with the triple-barrier method.","links":[{"text":"Level 3 shows how to do this better with the triple-barrier method","href":"https://alysly.com/learn/labeling/"}]},{"type":"heading","level":2,"id":"step-3-establish-the-base-rate","text":"Step 3: Establish the Base Rate"},{"type":"paragraph","text":"This is the most important step and the one most often skipped. Before you measure how often the pattern works, you have to know how often the same outcome happens without it."},{"type":"paragraph","text":"Suppose the price rises in 54 percent of all ten-period windows. A pattern with a hit rate of 56 percent is then noise, not a discovery. The hit rates published in pattern books almost never state a base rate – and without one, a percentage means nothing."},{"type":"heading","level":2,"id":"four-ways-to-fool-yourself","text":"Four Ways to Fool Yourself"},{"type":"paragraph","text":"Look-ahead bias. The calculation uses information that was not available at the time of the decision. The classic case is the close of the signal candle. It goes into the entry, although it is only known at the end of the period. The error is small, but it behaves like an edge."},{"type":"paragraph","text":"Survivorship bias. The sample only contains instruments that still exist today. Bankruptcies and delistings are missing – and with them exactly the cases where everything went wrong."},{"type":"paragraph","text":"A sample that is too small. Thirty occurrences support no statement about a hit rate. With 30 observations, the confidence interval around a measured 60 percent runs roughly from 41 to 77 percent. That range covers everything from useless to excellent."},{"type":"image","src":"https://alysly.com/img/ta/sample-size.391aa56b44166ff24d8e481ab478e51680e037d1aca6170f05b02f03d16267ff.webp","alt":"Funnel showing how the uncertainty around a measured hit rate shrinks as the number of cases grows"},{"type":"paragraph","text":"The same true hit rate of 53%, measured on different numbers of cases. (1) At 10 cases the plausible range runs from roughly 22% to 84% — any claim about the pattern would be a guess. (2) At 1000 cases it sits between 50% and 56%. (3) The dashed line is the true value, which you never get to see in practice. This funnel is why a pattern with 30 hits in a backtest proves nothing: it is not the pattern that is unclear, it is the measurement."},{"type":"paragraph","text":"Multiple testing. You try twenty variants, find a good one and report that one. This is the most expensive mistake in the whole discipline, and it is the reason Level 3 is almost entirely about it."},{"type":"heading","level":2,"id":"step-4-separate-development-and-testing","text":"Step 4: Separate Development and Testing"},{"type":"paragraph","text":"You develop the rule on one period and test it on another that you did not look at during development. That sounds obvious, and it is rarely done cleanly. The usual reason is that the test period gets used “one more time” after a disappointing first result. From the second look onwards it is no longer a test period."},{"type":"paragraph","text":"For price data, the simple random split that is common in machine learning is not enough: price series depend on time, and neighboring observations overlap. Level 3 covers why the whole procedure fails on that and how financial research repairs it.","links":[{"text":"Level 3","href":"https://alysly.com/learn/#level-3"}]},{"type":"heading","level":2,"id":"what-a-backtest-can-and-cannot-show","text":"What a Backtest Can and Cannot Show"},{"type":"paragraph","text":"It can show that a rule did not work in the past. That is a solid result, and the most common one."},{"type":"paragraph","text":"It cannot show that a rule will work in the future. At best, it shows that the rule worked during one particular period. The value of that result depends almost entirely on how many variants you tried before finding this one."},{"type":"paragraph","text":"To put a portfolio idea against historical data right away, use the backtester and the Monte Carlo tool. The second one shows the range of outcomes the same statistics could just as easily have produced, instead of a single path.","links":[{"text":"backtester","href":"https://alysly.com/tools/backtest/"},{"text":"Monte Carlo tool","href":"https://alysly.com/tools/monte-carlo/"}]},{"type":"heading","level":2,"id":"next","text":"Next"},{"type":"paragraph","text":"That is the end of Level 2. Level 3 picks up exactly this thread. It covers handing the testing to a machine and why standard machine learning fails on financial data."},{"type":"list","ordered":false,"items":[{"text":"Previous Measuring Volatility: Bollinger and ATR","href":"https://alysly.com/learn/volatility/"},{"text":"Next · Level 3 AI Trading: How It Works and Where It Fails","href":"https://alysly.com/learn/ai-trading/"}]},{"type":"heading","level":2,"text":"Indices"},{"type":"table","caption":"Indices, performance over the selected period","columns":["Index","1M"],"rows":[[{"text":"Dow Jones","href":"https://alysly.com/markets/indices/dow-jones/"},{"text":"+1.0%","data":{"m1":"+1.0%","m3":"-4.3%","m6":"+4.9%","ytd":"+7.6%","max":"+30.0%"}}],[{"text":"S&P 500","href":"https://alysly.com/markets/indices/sp500/"},{"text":"-2.6%","data":{"m1":"-2.6%","m3":"-2.2%","m6":"+4.2%","ytd":"+10.6%","max":"+35.6%"}}],[{"text":"MSCI World","href":"https://alysly.com/markets/indices/msci-world/"},{"text":"-2.0%","data":{"m1":"-2.0%","m3":"-2.4%","m6":"+3.7%","ytd":"+10.0%","max":"+35.0%"}}],[{"text":"MSCI ACWI","href":"https://alysly.com/markets/indices/msci-acwi/"},{"text":"-3.3%","data":{"m1":"-3.3%","m3":"-3.0%","m6":"+2.3%","ytd":"+10.2%","max":"+41.5%"}}]]},{"type":"heading","level":2,"text":"ETFs"},{"type":"table","caption":"ETFs, performance over the selected period","columns":["Fund","1M"],"rows":[[{"text":"SPDR S&P 500 ETF Trust SPY","links":[{"text":"SPDR S&P 500 ETF Trust","href":"https://alysly.com/markets/etfs/us/spy/"}]},{"text":"-2.6%","data":{"m1":"-2.6%","m3":"-2.2%","m6":"+4.2%","ytd":"+10.6%","max":"+35.6%"}}],[{"text":"Vanguard S&P 500 ETF VOO","links":[{"text":"Vanguard S&P 500 ETF","href":"https://alysly.com/markets/etfs/us/voo/"}]},{"text":"-2.6%","data":{"m1":"-2.6%","m3":"-2.2%","m6":"+4.2%","ytd":"+10.6%","max":"+34.1%"}}],[{"text":"iShares Core MSCI World UCITS ETF USD (Acc) EUNL","links":[{"text":"iShares Core MSCI World UCITS ETF USD (Acc)","href":"https://alysly.com/markets/etfs/ishares-core-msci-world-ucits-etf-usd-acc/"}]},{"text":"-2.0%","data":{"m1":"-2.0%","m3":"-2.3%","m6":"+4.3%","ytd":"+10.6%","max":"+487.5%"}}],[{"text":"Xtrackers MSCI Emerging Markets UCITS ETF 1c XMME","links":[{"text":"Xtrackers MSCI Emerging Markets UCITS ETF 1c","href":"https://alysly.com/markets/etfs/xtrackers-msci-emerging-markets-ucits-etf-1c/"}]},{"text":"-9.4%","data":{"m1":"-9.4%","m3":"-4.1%","m6":"-3.3%","ytd":"+14.0%","max":"+93.0%"}}]]},{"type":"heading","level":2,"text":"Stocks"},{"type":"table","caption":"Stocks, performance over the selected period","columns":["Stock","1M"],"rows":[[{"text":"NVIDIA Corporation NVDA","links":[{"text":"NVIDIA Corporation","href":"https://alysly.com/markets/stocks/nvidia-corporation/"},{"text":"United States","href":"https://alysly.com/markets/stocks/browse/?country=United%20States"}]},{"text":"-13.5%","data":{"m1":"-13.5%","m3":"-2.5%","m6":"+5.2%","ytd":"+10.8%","max":"+62.2%"}}],[{"text":"Apple Inc. AAPL","links":[{"text":"Apple Inc.","href":"https://alysly.com/markets/stocks/apple-inc/"},{"text":"United States","href":"https://alysly.com/markets/stocks/browse/?country=United%20States"}]},{"text":"-0.3%","data":{"m1":"-0.3%","m3":"+7.3%","m6":"+16.8%","ytd":"+22.1%","max":"+45.8%"}}],[{"text":"Tesla Inc. TSLA","links":[{"text":"Tesla Inc.","href":"https://alysly.com/markets/stocks/tesla-inc/"},{"text":"United States","href":"https://alysly.com/markets/stocks/browse/?country=United%20States"}]},{"text":"+5.2%","data":{"m1":"+5.2%","m3":"+21.7%","m6":"+2.3%","ytd":"-11.4%","max":"+65.3%"}}]]},{"type":"paragraph","text":"Data note 1","href":"#dn-1"},{"type":"heading","level":2,"text":"More Studies"},{"type":"list","ordered":false,"items":[{"text":"Jul 21, 2026 Collectibles ×5.4 for the same cards, matched-model index, Mar 2022 – Aug 2026 Is the Pokémon Card Market in a Bubble? 4,159 Sales Analyzed Prices for PSA-graded cards multiplied. Measured on the same cards they rose 438%, while the raw median shows 786% because pricier cards came up for sale later.","href":"https://alysly.com/analysis/pokemon-card-market-bubble/"},{"text":"May 19, 2026 Crypto 45.6% Bitcoin’s annualized volatility, Jul 2024 – Sep 2026 Bitcoin in Investment Portfolios: The Real Numbers Bitcoin is calmer than it was and still about three times as volatile as the stock market. It moves with stocks, not with gold.","href":"https://alysly.com/analysis/bitcoin-investment-portfolios/"},{"text":"Mar 17, 2026 Wealth €1.2m needed for €3,000 a month indefinitely at a 3% real return Living Off Your Returns: What It Really Takes The capital you need is arithmetic. The return you assume is a guess, and it can triple the answer.","href":"https://alysly.com/analysis/living-off-returns/"}]},{"type":"paragraph","text":"View all studies","href":"https://alysly.com/analysis/#studies"},{"type":"heading","level":2,"text":"Learn"},{"type":"list","ordered":false,"items":[{"text":"AI Trading: How It Works and Where It Fails From data to order: how AI trading works, and the hurdle it has to clear.","href":"https://alysly.com/learn/ai-trading/"},{"text":"Why Machine Learning Fails in Markets Why standard methods break on price series – the list of failure modes.","href":"https://alysly.com/learn/why-ml-fails-in-finance/"}]},{"type":"paragraph","text":"Course overview","href":"https://alysly.com/learn/"},{"type":"paragraph","text":"Updated Nov 1, 2026 · Content and figures checked Oct 19, 2026"},{"type":"heading","level":2,"id":"note-legend-h","text":"Data Notes"},{"type":"paragraph","text":"All calculations run automatically and are not investment advice."},{"type":"list","ordered":true,"items":[{"id":"dn-1","text":"Returns computed automatically from the price series, excluding dividends, taxes and costs."}]},{"type":"paragraph","text":"Data and methodology: Alysly · Responsible for content: Alexander Bock","links":[{"text":"Alysly","href":"https://alysly.com/methodology/"},{"text":"Alexander Bock","href":"https://alysly.com/about/"}]},{"type":"paragraph","text":"View all data notes","href":"https://alysly.com/notes/"}],"structured_data":[{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://alysly.com/#org","name":"Alysly","url":"https://alysly.com/","logo":{"@type":"ImageObject","url":"https://alysly.com/img/icon-512.png","width":512,"height":512},"description":"Data-driven insights, transparent analysis and powerful tools for long-term investors.","email":"mail@alysly.com","contactPoint":{"@type":"ContactPoint","contactType":"customer support","email":"mail@alysly.com","availableLanguage":["de","en"]},"address":{"@type":"PostalAddress","addressLocality":"Haltern am See","postalCode":"45721","addressCountry":"DE"},"founder":{"@id":"https://alysly.com/#founder"},"sameAs":["https://www.linkedin.com/company/alyslycom/","https://www.youtube.com/@alyslycom","https://x.com/alyslycom"],"publishingPrinciples":"https://alysly.com/methodology/"},{"@type":"Person","@id":"https://alysly.com/#founder","name":"Alexander Bock","jobTitle":"Founder and data analyst","url":"https://alysly.com/about/","sameAs":["https://www.linkedin.com/in/abxk/"]},{"@type":"WebSite","@id":"https://alysly.com/#site","url":"https://alysly.com/","name":"Alysly","alternateName":"Alysly Investment Analysis","inLanguage":"en-US","publisher":{"@id":"https://alysly.com/#org"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://alysly.com/markets/stocks/browse/?q={search_term_string}"},"query-input":"required name=search_term_string"}},{"@type":"WebPage","@id":"https://alysly.com/learn/testing-a-pattern/#webpage","url":"https://alysly.com/learn/testing-a-pattern/","name":"Testing a Pattern – Alysly","description":"Turning a pattern into a testable rule: base rate, sample size, look-ahead bias, survivorship bias and the question of how much a backtest really shows.","inLanguage":"en-US","isPartOf":{"@id":"https://alysly.com/#site"},"dateModified":"2026-11-01T18:26:04+01:00","publisher":{"@id":"https://alysly.com/#org"},"mainEntity":{"@id":"https://alysly.com/learn/testing-a-pattern/#entity"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://alysly.com/img/ta/sample-size-og.4fa36868dc3967294cfa82316ac98ba9f3b1fc27dae7120d9b3d4cafaa4d4fb3.png","width":1200,"height":630}}]},{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Investing & Trading Basics","item":"https://alysly.com/learn/"},{"@type":"ListItem","position":2,"name":"Testing a Pattern"}]},{"@context":"https://schema.org","@type":"Article","@id":"https://alysly.com/learn/testing-a-pattern/#entity","headline":"Testing a Pattern","description":"Turning a pattern into a testable rule: base rate, sample size, look-ahead bias, survivorship bias and the question of how much a backtest really shows.","inLanguage":"en-US","mainEntityOfPage":"https://alysly.com/learn/testing-a-pattern/","image":[{"@type":"ImageObject","height":630,"url":"https://alysly.com/img/ta/sample-size-og.4fa36868dc3967294cfa82316ac98ba9f3b1fc27dae7120d9b3d4cafaa4d4fb3.png","width":1200},{"@type":"ImageObject","height":500,"url":"https://alysly.com/img/ta/sample-size.391aa56b44166ff24d8e481ab478e51680e037d1aca6170f05b02f03d16267ff.webp","width":1280}],"isPartOf":{"@type":"CreativeWorkSeries","name":"Investing & Trading Basics","url":"https://alysly.com/learn/"},"educationalLevel":"Level 2","author":{"@id":"https://alysly.com/#founder"},"publisher":{"@id":"https://alysly.com/#org"},"datePublished":"2026-08-02T15:41:33+02:00","dateModified":"2026-11-01T18:26:04+01:00"}],"notice":"© 2026 Alysly.com. Not investment advice. All data without guarantee. Financial information is analyzed and presented partly with AI assistance."}