Time-Series Validation Tradeoffs: Training, Coverage, and Causality

Summary

The paper studies a fundamental conflict in time-series model validation: training runs need enough observations, test folds should cover enough of the sample, and training must precede each test point. It formalizes these aims through bounds relating training sufficiency, test coverage, future-data leakage, and the distance between test points and future training observations. The authors also bound leakage bias under a beta-mixing assumption, linking its size to temporal separation.

The analysis characterizes expanding walk-forward validation as the causal frontier and compares it with k-fold and purged k-fold schemes. The text says shuffled five-fold validation on pure noise produced an information coefficient of +0.32, while contiguous five-fold yielded +0.004 despite using the same amount of future data. These conclusions depend on the stated mathematical setup; the supplied summary does not provide proof details or broader empirical tests. It also cautions that an embargo can reduce leakage when dependence fades quickly, but cannot resolve causality problems tied to non-stationarity.

Key ideas

• Training sufficiency, test coverage, and temporal causality cannot all be maximized at once under the paper’s bounds.
• Validation that exceeds the causal frontier must use future observations in training.
• Leakage bias depends on the temporal distance to future training data under the stated beta-mixing assumption.
• Expanding walk-forward validation is presented as the causal frontier, while k-fold schemes trade causality for coverage.
• An embargo can reduce leakage when a process forgets quickly but cannot fix non-stationarity.

Tags

# 2609.29530


# The Impossible Trinity of Time-Series Validation: A Conservation Law among Training Sufficiency, Test Coverage, and Temporal Causality









Validating a model on a time series asks for three things at once: each training run should use most of the sample (sufficiency), the test sets should together cover most of the sample (coverage), and training data should come before test data (causality). We prove that the three cannot be had together and price each one. Let $α$ be the smallest training fraction over folds, $β$ the fraction of the sample covered by tests, $Λ$ the fraction of the sample used as training data from the future of a test point, and $δ$ the distance from a test point to the nearest training point in its future. Every scheme on a sample of length $T$ satisfies $α+β\le 1+Λ$ and $α+\min\{β,δ/T\} \le 1$, and under $β$-mixing the leakage bias at a test point is at most $2Mβ_{\mathrm{mix}}(δ)$. In words: going beyond the causal frontier $α+β=1$ requires training on the future; that future data must sit within $(1-α)T$ of a test point; and its harm depends on its distance, not its amount. Hence expanding walk-forward is exactly the Pareto frontier of causal validation, $k$-fold cross-validation buys the most future data, and purged $k$-fold with an embargo pays in distance instead, which is cheap when the process forgets quickly but cannot repair the part of causality demanded by non-stationarity. On pure noise, shuffled 5-fold reports an information coefficient of $+0.32$, while contiguous 5-fold, using the same amount of future data, reports $+0.004$.

Shown in full with attribution under the source's licence. Licence: abstract CC0

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.