Testing Whether In-Sample and Out-of-Sample Periods Differ
Summary
The document addresses whether results from one historical period used out of sample should match results from another period used in sample after the split is reversed. Its answer is no: different periods need not produce matching estimates or performance, and a match would be unexpected. The relevant empirical question is whether the periods can plausibly be treated as coming from the same population, rather than whether their results coincide.
It suggests comparing the periods’ distributions with a Kolmogorov–Smirnov test, or testing regression stability by adding a post-break dummy and interactions between that dummy and the regressors. An F-test can then assess whether the added terms significantly improve fit relative to a restricted model. The answer outlines a residual-sum-of-squares calculation and discusses conventional p-value thresholds, while cautioning that failure to reject a difference does not prove the periods are identical. These procedures depend on model specification and test assumptions; the document offers guidance rather than an analysis of actual data.
Key ideas
- Reversing which period is in sample does not imply that the resulting estimates or performance will match.
- The core question is whether the periods appear to come from the same population.
- A Kolmogorov–Smirnov test can compare sub-sample distributions.
- Regression stability can be assessed with period dummies and interactions, followed by an F-test.
- A non-significant result does not establish that the periods are identical.
Tags
Full text
# in-sample vs. out of sample conundrum # in-sample vs. out of sample conundrum - Lets say I for work with data from 2000-now in one sample (in-sample), and lets say that my out-of -sample will be from 2000 to 1950. I will then get some type of out-of-sample result. - If i then run a regression with in-sample data being from 1950, and up til 2000 (meaning that my-out of-sample will be from 2000-now), will i then get the same results from out-of-sample in point 1, as i will get in-sample in point 2? ## Answer by demully (score 1, accepted) https://quant.stackexchange.com/a/54274 No, you should not expect to get the same results. It would be a spooky coincidence if they actually coincided. Sufficiently spooky as to warrant an audit if they did! Rather the question you can ask is the likelihood the distributions of the two sub-samples are drawn from the same or a different population. Pooling the means and and the variances of both, ie looking at 1950-now, then what is the probability that both 1950-2000 and 2000-now come from the same wider population with these wider features? That’s a Kolmogorov-Smirnov test. Else - simpler -if you were simply interested in whether post-2000 looked different from 1950-2000, then add a dummy variable 0-1 for everything after 2000. And another interactive dummy = 1x your regressor variable after 2000 and 0 before. Or multiple dummies for each regresssor. Regress this for 1950-now with these dummies and without. Then F-test the restriction that pre-2000 looks the same as post-2000. Take the RSS of the two, and calculate the difference. Divide it by the RSS of the unrestricted, the lower value with with the dummies included. Multiply by (n-k), which is the number of datapoints (n) minus the variables in the model, including the intercept and all your dummies. Divide it by the number of dummies used. Call these “d”, This is your “F-statistic”. It’s a measure of whether adding all of those dummies was statistically significant or not. Excel or statistical tables will give you a p-value for F(n-k,d). This is the probability you’d randomly get the same results without trying to make post-2090 look different to pre-2000 By (subjective) convention, p below 5% suggests something is “maybe” different. Below 1%, “probably” and below 0.1% “almost certainly” so. The caveat being that a p-value of say 10% still means it’s more likely than not they differ. You just can’t hope to prove it. You cannot be certain they are not the same. Hope this helps.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.