Backtesting Expected Shortfall Without Direct Elicitability
Summary
The discussion distinguishes elicitation from backtesting. Expected Shortfall (ES) lacks a scoring function that uniquely elicits it as a point forecast, but that limitation does not by itself make ES impossible to backtest. The document cites work proposing multiple ES backtests, though it does not describe their procedures in detail.
One practical approach is to test several quantiles in the tail that contributes to ES. Quantile forecasts can be assessed through exceedance outcomes, and their results provide indirect evidence about the tail distribution underlying the ES estimate. The discussion notes that this evaluates the tail inputs rather than comparing the ES number directly with a realized value. It offers no worked examples or performance comparisons, so it leaves the choice and reliability of a particular test open.
Key ideas
- Expected Shortfall is not elicitable through a scoring function that uniquely evaluates its point forecast.
- Lack of elicitability does not rule out backtesting a risk forecast.
- Backtesting several tail quantiles can provide indirect evidence about the distribution used to calculate Expected Shortfall.
- Quantile tests assess the tail inputs rather than the Expected Shortfall value directly.
Tags
Full text
# ES not elicitable # ES not elicitable Expected Shortfall is not elicitable as some papers have pointed out. That simply means that there is no scoring function that elicits ES. My question is, does this imply that Expected Shortfall point forecasts are impossible to backtest? Still, there are backtests available for ES. Second question, how is the ES forecast backtested, that is how is the value of ES used in any backtest. To what is the ES forecast compared to in a backtest. ## Answer by RiskyScientist (score 4) https://quant.stackexchange.com/a/30086 I think it was T. Gneiting in 2011 who first proved that ES is not elicitable (Making and Evaluating Point Forecasts, Journal of the American Statistical Association Volume 106, 2011 - Issue 494) , which then threw some doubt as to whether it was backtestable. Carlo Acerbi pretty much put the matter to bed a few years ago in a number of papers, in which he explained that it does not matter for backtesting purposes whether or not ES is elicitable. Here is a link to his 2014 paper in which he gives three methods for backtesting ES. ## Answer by Kiwiakos (score 1) https://quant.stackexchange.com/a/24849 This is formally correct. However, I am not sure if practically it really makes any difference as Tasche points out: https://workspace.imperial.ac.uk/mathfin/Public/Seminars%202013-2014/Tasche_November2013_Slides.pdf Edit: ES can be expressed as a weighted average of percentiles, which are backtestable as Bernoulli. Therefore backtest a handful of quantiles and you effectively backtest ES (Tasche says four, I believe FRTB says two). The point is that you don't backtest the ES number directly, but quantiles of the tail that generated your ES number. Standard Deviation is not elicitable neither, but if you give me a sample I can tell you if the StdDev you gave me is good enough.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.