Choosing Directional Forecast Metrics When Returns Can Be Zero

Summary

The document raises a backtesting problem: mean directional accuracy compares the signs of predicted and realized moves, but realized zero moves do not fit neatly into that comparison. It also notes that weighted absolute percentage error is unsuitable when realized values can be negative, while mean squared error does not directly provide a percentage accuracy score.

No solution or evidence is provided; the text is a question seeking best practice. Its main lesson is that metric choice depends on how zero outcomes and signed values are defined and scored. A useful evaluation should state how ties or unchanged prices count, and should distinguish directional performance from the size of forecast errors. The document does not specify a forecasting horizon, asset class, or intended decision use, so it cannot establish a single preferred metric.

Key ideas

• Mean directional accuracy needs an explicit rule for realized moves equal to zero.
• Weighted absolute percentage error is problematic when realized values can be negative.
• Error metrics and directional hit rates measure different aspects of forecast quality.
• The document asks for guidance but does not supply or validate a recommended metric.

Tags

# Mean directional accuracy and zero


# Mean directional accuracy and zero












I'm trying to use mean directional accuracy to evaluate my directional predictions in back-test, but it can't deal with realised directions which are 0, due to the comparison of the signs of prediction and realisation.

I've also tried using WAPE, but this doesn't handle negative realisations.

I could use mean squared error, but I'd like to end up with a percentage value for accuracy.

What's the best practice?

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.