Level 3 AI in trading · Part 8/8

Feature Importance and Bet Sizing

Two closing questions that go beyond model quality: why does a model work? And how does its output become a position?

Why feature importance matters more than accuracy

At the signal-to-noise ratio of financial markets, a model’s accuracy says only so much. Two models with identical hit rates can rest on completely different features – one on an economically plausible quantity, the other on an artefact of data preparation.

Feature importance analysis is therefore not a post-processing step but the actual research output. It is the only place where a model explains something.

Three methods

Mean decrease impurity (MDI). For tree methods: how much did a feature improve splits across all trees? Fast, but with two known biases – features with many possible values are favoured, and the values refer only to training data.

Mean decrease accuracy (MDA). A feature’s values are shuffled after the fact and the drop in test performance is measured. It refers to unseen data and applies to any model. It must run on purged splits, otherwise you measure the leakage along with it.

Single feature importance. Each feature is evaluated alone as the sole predictor. This sidesteps the substitution effect entirely but misses features that only work in combination.

The substitution effect

The most important pitfall in interpretation. When two features are strongly correlated – say a 20- and a 25-period average – they share the importance. Both look mediocre although the underlying information is central. Conversely a single uncorrelated feature can look large without being especially valuable.

Level 2 already describes the same trap informally: five indicators from the same price series are not five independent opinions.

The usual handling: cluster correlated features first and measure importance per cluster. The answer is then no longer “feature 17 matters” but “the trend-measure cluster matters” – a statement that still holds tomorrow. How strongly market quantities move together can be read directly from the correlation matrix .

From probability to position

A classification model outputs a probability, not a quantity. The translation step co-determines the result.

Two principles:

Size proportional to confidence. Instead of trading every position at the same size, size is derived from the model probability – near zero around 50 per cent, growing with confidence. The common implementation maps the deviation from 50 per cent through a normal distribution function onto a value between 0 and 1.

The ceiling comes from risk, not from probability. Model confidence sets the share of the maximum position; the maximum itself comes from the risk calculation of Level 1 . A very confident signal does not justify a position that damages the account when it fails.

There is also a practical question that tends to vanish in backtests: when several signals run at once, their sizes have to be aggregated – otherwise many small, correlated positions add up to one large risk.

Closing the course

Three levels, one continuous thread: Level 1 describes what a chart shows. Level 2 makes the patterns precise enough to be testable. Level 3 shows what that testing looks like when taken seriously – and how many ways there are to fool yourself along the way.

The sober summary: most of the work in this field is not about finding edges but about ruling out apparent ones. Skip that part and you will always find something – and you will not keep it.

Tools on this site

Sources

The methods on this level come from the financial machine learning literature, in particular Marcos López de Prado: Advances in Financial Machine Learning (Wiley 2018) and the accompanying ORIE 5256 lecture series, The 7 Reasons Most Machine Learning Funds Fail, and The Probability of Backtest Overfitting (Bailey, Borwein, López de Prado, Zhu). The presentation, examples and graphics here are our own.