Validating Through-the-Cycle Credit Rating Probabilities
Summary
The document considers how to validate through-the-cycle (TTC) credit rating systems, which aim to represent default risk across changing economic conditions. It contrasts them with point-in-time (PIT) systems: PIT forecasts over a defined horizon can be compared with realized defaults over that horizon, while TTC probabilities may not align with observed rates in any single period because the current cycle affects defaults. It also distinguishes discrimination measures, such as ROC or Gini, from checks on whether predicted probabilities are calibrated.
The response suggests that regulatory attention often focuses on downturn loss-given-default estimates and that many practical rating systems are closer to PIT, making conventional validation easier to apply. It emphasizes that validation also involves analyst judgment and points readers toward research references. The answer does not provide a specific TTC calibration test or a detailed procedure for separating cycle effects from probability accuracy. Its observations are qualified by the author’s experience rather than supported with comparative data in the document.
Key ideas
- PIT default forecasts can be compared with observed defaults over their stated horizon.
- TTC probabilities are harder to backtest because realized default rates vary with the economic cycle.
- Discrimination measures assess ranking ability but do not by themselves establish probability calibration.
- The response describes validation as a mix of quantitative methods and expert judgment.
Tags
Full text
# Are there any well known methods of testing through-the-cycle rating systems? # Are there any well known methods of testing through-the-cycle rating systems? Rating systems, as defined by the Basel II Accord, can be classified into two broad types - through-the-cycle (TTC) or point-in-time (PIT) - and the probability of default predicted by such a system can usually have different interpretations. In practice, no rating system is purely TTC or PIT. One advantage of PIT systems are that they are easy to 'backtest' - if the PIT system predicts the 1 year probability of default, we can always "score" the obligors using the system 1 year back, and then compare the actual default rates against the predicted probability of default. Since TTC systems predict the probability of default over different economic cycles, it is hard to backtest such a system as the realized default rates may not match with the probability depending on where we are in the economic cycle. I am having a hard time finding any existing research or literature on this topic. Most of them mention the standard measures like ROC, Gini, Accuracy Ratio, etc. which tells us whether the rating system can sufficiently differentiate the likelihood of default. However, this does not say if the probabilities output by the rating system make sense. Can anyone point me out to any relevant research, books or documents which may help? ## Answer by bVs (score 4) https://quant.stackexchange.com/a/7153 I think the national regulators are more concerned with downturn LGD (sort of TTC LGD) rather than a TTC PD. Therefore most rating systems which I encounter are closer to being PIT and thereby easier to validate using the techniques you mentioned and also to backtest. But in any case, model validation is a very subjective field despite the various quantitative tools that exist and the expert judgement of the analyst must come through. Anyway, some of the best research on model validation remains as follows: http://www.bis.org/publ/bcbs_wp14.htm http://www.risk.net/digital_assets/5028/jrm_v1n1a4.pdf I will add more references as I track their urls ;)
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.