Online Smoothing and Change Detection for Time Series
Summary
The document considers how to adapt an exponential smoothing parameter online, distinguishing a univariate estimator such as a mean or variance from multivariate regression. It argues that choosing a smoothing window is meaningful only when the data may change over time: under an independent, identically distributed assumption, the usual empirical mean uses all observations and its estimation variance falls as the sample grows. A shorter window remains unbiased in that setting but has greater, persistent variance.
For changing regimes, the answer suggests detecting shifts rather than treating one fixed smoothing rate as universally optimal. A generic approach is to define a contrast that tends to rise when the process changes, compare it with a threshold, and reset the smoother after a detected change; exponential averaging can be used between resets. It gives a simple example comparing recent and older window means, scaled by the older window’s standard deviation. The proposal is conceptual: it specifies no threshold, window calibration, false-alarm tradeoff, or empirical performance, and says regression adaptation calls for different tools such as state-space methods.
Key ideas
- A fixed smoothing rate is not automatically optimal for both scalar estimators and regression models.
- With independent, identically distributed observations, using more data improves mean precision over time.
- Shorter windows trade increased estimator variance for reduced dependence on distant observations.
- Regime changes can be handled by monitoring a contrast and resetting the smoother when a threshold is crossed.
- The suggested mean-shift contrast is illustrative and requires choices about windows and thresholds.
Tags
Full text
# Online algorithm for selecting smoothing parameter?
# Online algorithm for selecting smoothing parameter?
In Online Algorithms in High-frequency Trading the authors demonstrate online, exponentially-weighted algorithms for mean, variance, and linear regression.
The authors estimate their smoothing parameter $\alpha$ in-sample, but state "Another approach would be to estimate the optimal alpha online as well."
What would be an example of how to estimate the smoothing parameter online?
## Answer by lehalle (score 2, accepted)
https://quant.stackexchange.com/a/16577
First of all, I do not believe the "optimal smoothing" of an estimator (like the mean or the variance) and the "regression case" are the same.
The smoothing of an existing estimator (like mean or variance in the blog post), is an univariate problem, where the regression is a multivariate one. In the regression case, you should be able to change the coefficients of the whole formula when you adjust the smoothing, but in the "simple" smoothing one you are not expected to change the formula.
I will answer to the first case here. For the regression case, you should have a look at ARIMAX models and Kalman filters, it is (according to me) difficult here to split in two the process of (1) obtaining the good model, (2) apply a smoothing.
Obtaining an adequate filtering window size once you have a formula (say the formula of the mean), is already difficult. Say you observe $X_1,X_2,\ldots,X_k,\ldots$ at time $\tau_1,\tau_2,\ldots,\tau_k,\ldots$. You have to go back to the formula of the mean, where does it come from?
It is easy to see that the usual mean is the solution to the following minimization problem:
$$\min_m \mathbb{E}(m-X)^2.$$
It minimizes the expected variance between the observed random variable and the "mean". Under simple assumptions ($X$ is i.i.d.) $m$ is the empirical mean, using as much data as possible. In terms of smoothing it would mean you take all the points since the start of the day (I uses your intraday example), and averages more data with time:
$$m_\infty(X_k,k\leq K)=\frac{1}{K} \sum_{k\leq K} X_k.$$
And the variance of this estimator is proportional to $\sqrt{\mathbb{V}(X)/K}$.
Why would you need something different?
If you want to take less points, for instance a sliding window of size $W$:
$$m_W(X_k,k\leq K)=\frac{1}{W} \sum_{k= K-W+1}^K X_k,$$
you will obtain again an unbiased estimator, but with a largest variance, since you replace $\sqrt{\mathbb{V}(X)/K}$, where $K$ goes to infinity with time, by $\sqrt{\mathbb{V}(X)/W}$, that is constant.
If you do that it is because you believe $X$ is not i.i.d. (independent and identically distributed). In such a case you can have $X$ being in fact drawn from a random variable until time $T$, and then it switches to another distribution.
It means that you need to detects this change, or at least to be not too much sensitive to it. If you take an exponential average you are in the second case: you do not try to know anything about the switches, you just implement a formula such that after a while you do not take the past into account at all, and you weight more the recent past that an older one.
But you can do better than that. And there is a large literature on the kind of problem, called **change detection*. Here is a good review: State-of-the-Art in Sequential Change-Point Detection. Shiryaev wrote a lot of good papers on this topic.
The generic methodology is to choose a contrast (i.e. a criterion that will be low when the state remains the same, and high during the change), and to threshold it to detect a change of regime. Then it is enough to reinit your smoothing. And practically to be confortable you can use an exponential moving average between two switches if you want.
For the mean, a typical and simple contrast is the recent empirical mean vs. an old one normalized by a standard deviation:
$$C(K):=\left|\frac{{\rm mean}(X_{K}:X_{K-h})-{\rm mean}(X_{K-2h}:X_{K-h})}{{\rm std}(X_{K-2h}:X_{K-h})}\right|,$$
with the notation $X_{a}:X_{b}$ the time series from $b$ to $a$.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.