Backtesting Quantile Forecasts for Discrete Outcomes
Summary
The document asks how to backtest a forecast of a percentile when the underlying observations are discrete, using the number of days between credit-card billing and payment as an example. It contrasts this with a continuous default-rate example, where a rolling historical sample supplies a 95th-percentile forecast and observations above that threshold count as exceptions.
For the continuous case, it describes checking whether exceptions occur at the expected frequency with a proportion-of-failures test, and whether they cluster with an independence test. It cites Kupiec and Christoffersen and gives illustrative exception-count bands from the former. The proposed discrete-outcome method itself is not explained: the accepted response points to a paper on discrete quantile estimation. Thus, the document identifies the modeling and validation question but does not provide enough detail to implement or assess that method; the continuous-case procedure is background, not a demonstrated solution for discrete quantiles.
Key ideas
- A percentile forecast can be defined for discrete observations such as payment delays.
- The example uses a rolling historical window to estimate a high quantile of monthly default rates.
- Exception frequency can be assessed with a proportion-of-failures test.
- Exception dependence can be examined with an independence test.
- The document points to research on discrete quantile estimation but does not describe its procedure.
Tags
Full text
# backtesting a 5% quantile model of a discrete value random variable?
# backtesting a 5% quantile model of a discrete value random variable?
If a random variable is discrete, and we are interested in its quantile value, how to define a proper back testing procedure?
For example, the underlying variable with a discrete value is
$$ d(\mbox{account}) = \mbox{PaymentDate} - \mbox{BillingDate} $$
the observing variable:
$$ y = \mbox{percentile}(d, 95\%, \mbox{month}) $$
or $y$ is the 95th percentile value of $d$, for a particular month. e.g. 95% of credit cards are paid within 20 days from the billing, in 2013 Jan.
How could I define a back-testing approach?
## Background
To define an estimation-backtesting method for a continous random variable is easier. Now in my group we have such a non-parametric approach:
underlying variable:
$$r(\mbox{month}) = \mbox{monthly credit-card account default rate}$$
For example, 2013 Feb default rate is 1.1%, 2013 Jan is 1.2%...
observing variable:
$$ x = \mbox{percentile}(r, 95\%) $$
$x$ is the 95% percentile value of $r$. Here $x$ definition is similar to VaR.
point forcast:
$$ \hat x(\mbox{month}) = \mbox{percentile}(r(\mbox{month}), N, 95\%) $$
$\hat x$ is the 95% percentile value of $r$, based on $N$ historic observations.
For example, take $N=36$, retrieve back 36 months, the 95% percentile value of default rate $r$ is 2.3%. then $\hat x = 2.3\%$.
point forecast Exception:
$$ \mbox{PFException}(t) = \begin{cases} 0 & r(\mbox{month}) \leq \hat x(\mbox{month}) \\ 1 & \text{otherwise} \end{cases} $$
By right 95% of the time there shall have no exception, while 5% of the time exception happens.
backtesting:
There are POF test, checking the rate of the exception; and independent test, checking the correlation of exceptions.
For example, Kupiec (1995) proposed a POF test checks exceptions happened in previouis 36 months' point forecasts: 0-4 exceptions are ok, green light, 4-7 exceptions are yellow light, while more than 8 exceptions are red light.
Christoffersen (1998) proposed an independent test.
Kupiec, P. (1995). Techniques for verifying the accuracy of risk management models. Journal of Derivatives 3, 73–84.
Christoffersen, P. (1998). Evaluating interval forecasts. International Economic Review 39, 841–62.
## Answer by athos (score 1, accepted)
https://quant.stackexchange.com/a/7511
Just to answer my own question. Discrete variates' quantile, could be modelled and tested. I followed this paper: "Discrete Quantile Estimation", Halina Frydman and Gary Simon, New York University, January 29, 2007. Detailed approach is inside.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.