Ranking Probabilistic Forecasts by Quantile Coverage for Trading

Summary

This study proposes statistical metrics for ranking probabilistic forecasting models, using pinball loss and empirical coverage. It asks whether these measures can identify the best-performing forecast model and whether the ranking metric is associated with higher average profit per trade out of sample. The focus is on forecasts expressed as quantiles and their use within a trading strategy.

In the reported comparison, ranking models by the coverage of quantile forecasts during the hours when trades are made produces better economic performance than the other considered metrics. This suggests that forecast calibration during relevant trading periods may matter more for the strategy than an aggregate measure alone. The finding is limited to the trading strategy and evaluation described; the document provides no market, sample length, numerical returns, or detailed comparison results. It therefore offers a selection principle to investigate, rather than evidence that coverage ranking will be superior for every forecasting task or trading system.

Key ideas

• The study compares probabilistic forecast rankings based on pinball loss and empirical coverage.
• Its evaluation links model rankings to average per-trade profit in an out-of-sample period.
• Coverage of quantile forecasts during trading hours yields the strongest economic performance in the tested strategy.
• The result is specific to the considered forecasts and trading setup.

Tags

# Ranking probabilistic forecasting models with different loss functions


# Ranking probabilistic forecasting models with different loss functions









In this study, we introduced various statistical performance metrics, based on the pinball loss and the empirical coverage, for the ranking of probabilistic forecasting models. We tested the ability of the proposed metrics to determine the top performing forecasting model and investigated the use of which metric corresponds to the highest average per-trade profit in the out-of-sample period. Our findings show that for the considered trading strategy, ranking the forecasting models according to the coverage of quantile forecasts used in the trading hours exhibits a superior economic performance.

Shown in full with attribution under the source's licence. Licence: abstract CC0

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.