How VaR Backtesting Differs from Model Validation
Summary
The document distinguishes ongoing model backtesting from broader model validation, using value at risk as its main example. Backtesting compares model forecasts with subsequently realized outcomes, while respecting the information that would have been available when the forecasts were made. For VaR, this includes checking whether portfolio losses exceed predicted limits and investigating whether the frequency of breaches matches expectations. Such monitoring can support performance assessment and help identify inaccurate components.
Validation examines whether a model is fit for its intended use. It can include backtesting, but also assesses conceptual soundness, methodological choices, sensitivity to data and parameters, implementation quality, documentation, and comparison with independent implementations or other models. The answers describe validation at implementation and periodic review, with ongoing monitoring in production. Some checks can serve both purposes. The distinction is practical rather than absolute: a complex, assumption-dependent backtesting method may itself warrant validation, and the appropriate review depends on the model and its use.
Key ideas
- Backtesting evaluates model performance by comparing forecasts with outcomes observed later.
- VaR backtesting reviews the frequency and causes of losses that exceed predicted risk limits.
- Model validation is broader and assesses whether a model’s logic, methods, inputs, code, and documentation suit its purpose.
- Validation commonly occurs before use and periodically, while performance monitoring continues during operation.
- A quantitative backtesting method may itself need validation if its design relies on substantive assumptions.
Tags
Full text
# Backtesting of Risk models
# Backtesting of Risk models
I am wondering if there is any difference between 2 terms call `model backtesting` and `model validation` from the perspective of `financial risk models` like `VaR`.
Could someone share some insight on similarity and dissimilarity of these 2 terms?
## Answer by ir7 (score 4)
https://quant.stackexchange.com/a/55854
Just adding the time/frequency dimension difference to what was said above: model backtesting is a model performance technique which takes place on an ongoing basis (in particular for VaR, breaches need to be discussed as soon as feasible after they take place; if there are too many over some period of time, they need to be escalated). Model validation takes place when the model first gets implemented and then every few years gets revalidated.
If you think the methodology (conceptual approach, statistical facts used in it) or implementation of the model backtesting are non-trivial (non-mechanical, quantitative, assumption based), one could classify it as 'model' and could undergo validation itself.
## Answer by Bob Jansen (score 2)
https://quant.stackexchange.com/a/55850
In my opinion model validation is broader than model backtesting. During model backtesting you test model performance on data that has been realised using only the data you could have used when using the model for risk management. This gives you an idea on model accuracy and it allows you to find out where the model or submodels are inaccurate or misleading.
Model validation can include backtesting and also takes a deeper look at the model itself. Things to look at during model validation:
- Does the logic make sense
- Are the methods used appropriate, do other more appropriate methods exist?
- How sensitive is the model to changes in parameters and data
- Is the model code correct and does it handle all edge cases well (or at least flags them for attention)
and any other thing that convince management and yourself that the model can be used for its intended purpose.
## Answer by Dimitri Vulis (score 2)
https://quant.stackexchange.com/a/55852
"Validation" means that someone analyses the model and pronounces it fit to be used, usually subject to conditions such as ongoing performance monitoring, and restrictions on input. Good industry practices include 1 for the validator to be independent from the model developer 2 to review and re-validate important models periodically.
"Ongoing performance monitoring" means that the performance of the model in production and/or on some regression test cases is monitored for signs of deterioration.
Some tools are used both in "validation" and in "OPM".
VaR "backtesting" - since VaR predicts that a book will lose no more than $\\\$X$ on some number of days, it's easy to compare the VaR predictions with the "hypothetical profit and loss" (without fees and other cash flows that VaR does not take into account). If the number of days when the book loses more than VaR ("VaR breaches") is very different from what VaR calculation predicted, then this needs to be analyzed and explained. This exercise is usually done both for VaR validation and as part of VaR OPM.
(Other examples of tests done both during validation and OPM may be - benchmarking the model outcomes against observations and the outcomes of other models ; testing that the greeks from pricing models plugged into Taylor expansion explain profit and loss.
An example of a test usually done only during validation is re-implementing the same model independenly and comparing the outcomes. Other validation tasks might include checking that the model documentation is OK (describes the model in enough detail that someone not familiar with the model might reproduce it; explains choices made by the modeler and alternative approaches). The tasks in the periodic review (not a full-blown validation) might include reviewing the OPM results and checking that the OPM plan still makes sense; checking published research and industry practices to see if the model needs revalidation.)Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.