Skip to content
All library documents

Why Lower Prediction Error May Produce Lower Trading Profit

Article Quant Q&A · Author: Saber
Read the original

Summary

The discussion explains why a machine-learning model with lower mean squared or root mean squared prediction error can still earn less in a backtest. Prediction accuracy evaluates numerical closeness to observed returns, whereas profit depends on how forecasts are converted into trades, including direction, sizing, and the strategy’s exposure rules.

An example compares two positive return forecasts when the realized return falls between them. A strategy that scales position size with forecast magnitude takes a larger position from the higher, less accurate prediction and can therefore earn more in that instance. This illustrates the possible mismatch between statistical loss and trading outcomes; it does not show that the less accurate model is generally more profitable. Model selection should reflect the intended trading objective and account for the strategy used to turn signals into positions.

Key ideas

  • Prediction error and trading profit measure different outcomes.
  • A strategy that sizes positions by forecast magnitude can trade more aggressively on a less accurate prediction.
  • The relationship between model accuracy and profit depends on the trading rules.
  • A single illustrative example cannot establish which model will perform better across markets or periods.

Tags

Machine learningEquitiesBacktestingPosition sizing
Full text
# Lower MSE results in less profit when using Machine Learning


# Lower MSE results in less profit when using Machine Learning












When using Machine Learning for predicting stocks, can a lower Mean Squared Error result in less profit after Backtesting or is there a mistake in the experiment?

## Answer by babelproofreader (score 1, accepted)

https://quant.stackexchange.com/a/45391

My answer to a similar question on the Cross Validated forum link here might be useful. In a nutshell you need to optimise for profit and not MSE - the two are not necessarily one and the same.

## Answer by alexprice (score 1)

https://quant.stackexchange.com/a/45397

It can , it depends on your trading strategy.

Let's say model1 predicts relative return as 0.1 for next day and model2 predicts relative return as 0.3, while actual return is 0.15.

Model1 has lower RMSE error.

If your trading strategy is to buy when model predicts positive next day return,and do so in volume proportional to model prediction, then you would buy more asset in model2 case, even if accuracy is lower. but you will make more money in this case. i.e. PNL of model1 would be less than PNL of model2.

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.