Level 3 AI in trading · Part 1/8
Why Machine Learning Fails in Markets
Machine learning succeeds in many fields because assumptions hold there that are violated in financial markets. Applying the methods unchanged to price data reliably produces models that look brilliant in hindsight and deliver nothing in production.
In structure, this level follows the research literature on financial machine learning, in particular the work of Marcos López de Prado (Advances in Financial Machine Learning, the ORIE 5256 lecture series, and The 7 Reasons Most Machine Learning Funds Fail). The methods are explained here in our own words and with our own graphics.
The starting position
An image model works with independent examples, clean labels and a stable relationship: a cat stays a cat. Financial data offers none of that.
- Observations are not independent. Consecutive prices are strongly correlated and overlapping windows share the same information.
- The signal-to-noise ratio is extremely low. A good edge is a few percentage points above chance, not 95 per cent accuracy.
- The relationship changes. What works gets traded and thereby disappears. That is not a data problem but the mechanism of the market.
- There is only one history. You cannot generate a second 2008. Every test runs on the same dataset everyone else uses.
The typical failure modes
The lone operator. One person is meant to source data, build features, train models, run backtests and trade. In any other quantitative discipline that would be unthinkable. The alternative is a production-line division of labour – data curation, feature analysis, strategy construction, backtesting – in which nobody carries an idea alone all the way to deployment.
The backtest as a research tool. If the backtest is used to find a strategy rather than to check one, the result is worthless. Every iteration consumes test data. After a hundred runs the test period is entirely used up and nobody noticed.
Chronologically invalid splits. Standard random cross-validation assumes independent observations. On price series it leaks future information into training – covered in detail under cross-validation .
Fixed time windows as a sample. One data point per hour treats a quiet night hour like the hour of a central bank decision. The resulting returns are strongly non-normal – see data and bars .
Bad labels. “Is price higher in ten days?” ignores what happened in between. A path that dips far below any stop is a loss in practice and a win in the label.
Ignored overlap. When a label reaches ten periods into the future, neighbouring examples overlap by 90 per cent. A model treating them as independent mistakes repetition for confirmation.
An unreported number of trials. Testing twenty variants and reporting the best means reporting a maximum. How badly that distorts the metrics is covered under backtest overfitting .
What follows
No single trick fixes this. The remaining articles of this level each address one failure mode and the method research developed against it:
| Problem | Method |
|---|---|
| Uneven information density | activity-based bars |
| Trend vs. stationarity | fractional differentiation |
| Unusable labels | triple barrier, meta-labeling |
| Overlapping examples | sample weights, uniqueness |
| Leakage during validation | purging and embargo |
| Too many trials | deflated Sharpe ratio, PBO |
One realistic expectation: none of these methods turns a bad result into a good one. They make it visible that it is bad – before real money rides on it. That is the actual gain.
Next
Starting with the foundation: how to turn a price series into a usable sample at all.