Level 2 Patterns & indicators · Part 9/9

Testing a Pattern

Up to here this course has been descriptive. This article is the transition: it turns an observation into a statement that can be wrong.

Step 1: make the rule unambiguous

A pattern is testable only when a program can find it without judgement. “A hammer after a downtrend” is not enough. A testable version would be:

Lower wick at least twice the body, upper wick at most a quarter of the body, and the close below the 20-period average.

Every number in that definition is a decision. Those decisions are the problem later – but without them there is nothing to measure.

Step 2: define the outcome

What exactly is being measured? “Price rises afterwards” is not a definition. Three things are needed: a horizon, a threshold and a comparison. For instance: is the close 10 periods later at least 1 ATR higher?

A worse variant is surprisingly widespread: price is checked after a fixed horizon. A path that plunges far below any sensible stop in between and recovers at the end counts as a success – even though it would have been stopped out in practice. How to do better is shown in Level 3 with the triple-barrier method .

Step 3: establish the base rate

The most important and most frequently skipped step. Before measuring how often the pattern works, you must know how often the same outcome occurs without it.

If price rises in 54 per cent of all ten-period windows, then a pattern with a 56 per cent hit rate is not a discovery but noise. Published hit rates in pattern books almost never state a base rate – without one, a percentage is meaningless.

Four ways to fool yourself

Look-ahead bias. The calculation uses information unavailable at decision time. The classic: the signal candle’s close feeds into the entry although it is only known at the period’s end. The error is small but behaves like an edge.

Survivorship bias. The sample contains only instruments that still exist today. Bankruptcies and delistings are missing – and with them exactly the cases where everything went wrong.

Too small a sample. Thirty occurrences support no statement about a hit rate. At 30 observations the confidence interval around a measured 60 per cent runs roughly from 41 to 77 per cent – a range containing everything from useless to excellent.

Multiple testing. You try twenty variants, find a good one and report that. This is the most expensive mistake in the whole discipline and the reason Level 3 is almost entirely about it.

Step 4: separate development and testing

The rule is developed on one period and tested on another that was not looked at during development. That sounds obvious and is rarely done cleanly – usually because the test period gets used “one more time” after a disappointing first result. From the second look onward it is no longer a test period.

For price data, the simple random split common in machine learning is not enough: price series are time-dependent and neighbouring observations overlap. Why the entire procedure fails on that and how financial research repairs it is the subject of Level 3 .

What a backtest can and cannot show

It can show that a rule did not work in the past. That is a solid result and the most common one.

It cannot show that it will work in future. At best it shows that it did during a particular period – and how much that result is worth depends almost entirely on how many variants were tried before this one was found.

To put a portfolio idea against historical data right away, the backtester and the Monte Carlo tool are the tools for it – the latter showing, instead of a single path, the range of outcomes that the same statistics could equally have produced.

Next

End of Level 2. Level 3 picks up exactly this thread: what happens when the testing is handed to a machine – and why standard machine learning fails on financial data.