Skip to content
All library documents

Evaluating Historical Models Under Structural Change

Article Quant Q&A · Author: athos
Read the original

Summary

The document considers how to defend a historical simulation model when external changes, such as a financial crisis or tax policy shift, cause its predictions to diverge from observed outcomes. The example is a non-performing loan rate model that may lag after a regime change and later recover as newer data enters the history.

The responses emphasize documenting model assumptions and identifying conditions under which they may fail. They mention possible diagnostics such as Granger causality, entropy or divergence measures, fit statistics, and information criteria, as well as error-correction models when deviations from a long-run relationship are plausible. For severe scenarios, stress testing is suggested alongside historical simulation. These are candidate tools, not a universal tolerance threshold: the document gives no standard acceptance criterion or detailed implementation, and notes that suitable evaluation depends on the model and its use.

Key ideas

  • A historical model depends on assumptions that can break when market or economic conditions shift.
  • Model documentation should state the assumptions and conditions under which they may fail.
  • Time-series diagnostics and fit measures can help assess changing relationships and model mismatch.
  • Stress tests can supplement historical simulation for crisis scenarios.
  • The document offers no universal threshold for acceptable model failure.

Tags

StatisticsRisk managementBacktesting
Full text
# How to justify a model that could not predict external factors?


# How to justify a model that could not predict external factors?












I'm building some models, for example, Bad Loan (NPL) rate.

It's based on historical simulation method -- basically it's saying the future behavior could be predicted by history data.

However, this is not always true, when the market changes. For example, when 2008 financial crisis came, NPL rates went up; when there's a tax cut, the NPL drops.

So, whenever market changes, the model will fail, but after a few months, it will pick up the trend and work again.

Now, how could I defend my model? How to justify that the failure is acceptable? Is there some industry standard / criterion about the tolerance of model mismatch caused by external factors (e.g. market, economics)?

## Answer by gamerx (score 3, accepted)

https://quant.stackexchange.com/a/7583

Well, typically in the process of coming up with a model you are supposed to understand the assumptions that you're making and the circumstances(preferably quantifiable) under which your assumptions will hold/break. No model is infallible and it is how well the assumptions are stated and understood that will determine if your model is acceptable.

I can't really provide any specific suggestions since I don't know about the fine details of what you're doing. But I'd looking into applying some form of granger causality metric if time-series are involved or to measure the non-stationarity of the data(entropy and f-divergences).

## Answer by 4pie0 (score 2)

https://quant.stackexchange.com/a/7582

it was meant to be a comment, so please don't treat it like an answer, just suggestion.

I think every department has its own standards. and if you want to constitute your model somehow, then you can just compute R^2 between real and fitted values, RMSD, information capacity criteria (AIC/BIC) or you can use any from tens other measures. you can also state that there is some long-run relationship, and these deviations are just short term deviations from the long-run equilibrium, maybe ECM (error correction) model is appropriate.

## Answer by Richi Wa (score 1)

https://quant.stackexchange.com/a/7585

I think most models failed in the 2008 crisis. Historical simulation and e.g. a value-at-risk calculated from it is designed for normal to "medium" market behavior.

To account for crisis scenarios stress tests should be in place. This what is done e.g. in the USCITS framework.

However, after the crisis your model should keep this history "in mind".

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.