Comparing Price Forecasts with the Diebold–Mariano Test

Summary

The document considers how to compare competing price forecasts when historical forecasts and realized prices are available. The questioner already calculates error means, standard deviations, upper percentiles, skewness, and corresponding statistics for absolute errors, then asks which further procedures can help select forecasting systems.

The accepted response recommends the Diebold–Mariano test as a way to assess whether two forecast systems differ in predictive accuracy using their forecasts and the observed outcomes. It describes the test as requiring no sophisticated assumptions about the models or processes, but gives no implementation details, worked example, or discussion of practical complications such as the chosen loss function or forecast horizon. The reference is to the original 1995 paper, though the response does not reproduce the paper itself.

Key ideas

• Error summaries describe forecast performance but do not by themselves test whether two systems differ significantly.
• The Diebold–Mariano test is suggested for comparing predictive accuracy from two forecast series and realized outcomes.
• The response characterizes the test as usable without sophisticated assumptions about the forecasting models or processes.
• The document does not explain implementation choices or limitations in detail.

Tags

# What should basic statistical analysis of different price forecasts contain


# What should basic statistical analysis of different price forecasts contain












we have set of historical data of different price forecasts and the real prices.

When assessing "which forecasts are best", what parts of the analysis should one never miss out? We are at a very start of our investigation, using forecast error mean, forecast error standard deviation, forecast error 90% percentiles, error distribution skewness. We calculate the same quantities for absolute values of the errors.

What quantities might we miss out? Are some procedures crucial for determining what price forecast systems to utilize? Are there any practical sources on such analysis?

## Answer by mark leeds (score 3, accepted)

https://quant.stackexchange.com/a/53349

Hi: Based on your question, it sounds like the Diebold-Mariano test might be perfect for your case. It doesn't require any sophisticated assumptions about models or processes etc. All one needs are the two sets of forecasts and the actuals. I can't find the actual paper but below is the reference to it. I imagine that, if you google hard enough, the paper itself is probably somewhere on the internet ( besides from the publisher ).

https://amstat.tandfonline.com/doi/abs/10.1080/07350015.1995.10524599#.XpjzqvkpDCI

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.