Level 3 AI in trading · Part 6/8

Cross-Validation Without Leakage

Cross-validation is the standard tool for checking whether a model generalises. On financial data, in its usual form, it systematically returns results that are too good.

Why k-fold leaks

Standard k-fold cross-validation splits the data randomly into k blocks, trains on k−1 and tests on the remainder. It assumes independence.

On price data that creates two leaks:

Through labels. A training example shortly before the test block has a label that reaches into the test block. Its outcome is determined by the same price moves as the test examples. The model has already seen part of the answer.

Through autocorrelation. Prices at neighbouring times are similar. A model that knows Monday’s and Wednesday’s price can nearly guess Tuesday without having learned anything economic.

The result is always the same: excellent validation scores and a disappointing live run.

Purging and embargo

Purging removes from training every example whose label window overlaps the test window. That ends the leak through labels.

Embargo additionally blocks a short span immediately after the test window. The reason is autocorrelation: examples right after the test still contain information about its end, even when their label windows do not formally overlap. A common size is one to five per cent of the data length.

Together they cost training data – with ten blocks, easily a mid single-digit percentage. That loss is the price of the metric meaning anything at all.

Why a single test period is not enough

Even with purging, a problem remains: one split path yields exactly one number. Whether it is stable or depends on a lucky cut point cannot be seen from it.

Combinatorial purged cross-validation addresses this: instead of one test block, several blocks are used as test sets in different combinations. This produces many distinct train/test paths and therefore a distribution of results rather than a single figure.

The benefit is the same as with the Monte Carlo tool for portfolios: you see the range of outcomes the same data could equally have produced. A strategy whose Sharpe ratio swings between 0.2 and 1.8 depending on the split is a different animal from one steady at 0.9 – even when both average the same.

What remains

Even a cleanly purged procedure only answers whether the model generalises on this data. It does not answer whether the relationship still exists tomorrow, and it does not correct for the number of variants tried.

That second question is the last and largest error – and the subject of the next article.

Next

How many attempts are hiding inside a good backtest?