Choosing Regression Metrics to Evaluate ONNX Price Forecasts

Summary

This article explains how regression metrics assess model predictions and how they differ from loss functions used during training. It reviews MAE, MSE, RMSE, R-squared, MAPE, MSPE, and RMSLE, describing their units, sensitivity to large errors, relative-error interpretation, and restrictions. It recommends R-squared for price prediction and using MAE, RMSE, and MAPE together to compare models.

The example evaluates four ONNX models forecasting EURUSD daily closes from prior bars, then compares selected models using regression metrics and Strategy Tester results. The article reports that one model performed better than another in both comparisons, while a weaker model also showed poor test results. It cautions that the example is a demonstration rather than a real-account trading system or statistical study: the sample contains only 22 values, which is too small for serious research.

Key ideas

• Loss functions support model optimization, while regression metrics assess prediction quality externally.
• MAE reports average absolute error in target units, while MSE and RMSE give greater influence to large errors.
• MAPE emphasizes relative error, and RMSLE requires nonnegative actual and predicted values.
• Comparing MAE with RMSE can indicate how widely individual errors vary.
• The ONNX example compares model metrics with tester results but uses too few observations for rigorous conclusions.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.