Online Mean and Variance Updates for Streaming Trading Data

Summary

This document derives recursive updates for the arithmetic mean and variance, so a process can incorporate each new observation without retaining the full history. It then presents exponentially weighted mean and variance updates, which give recent observations more influence and suit streaming indicators such as moving averages and volatility estimates. The article includes a Python implementation and discusses using incremental state for other statistics, including correlations and linear fits.

It also relates an EMA smoothing factor to an SMA window by matching their time centroids, and gives a formula for adjusting the factor when the update frequency changes. These are approximations whose behavior depends on initialization, sampling cadence, and the chosen decay rate. The supplied example is illustrative rather than a trading test, and the article does not compare forecast quality or returns. A fixed-decay EMA also differs from a rolling window: it retains a fading influence from older observations rather than dropping them at a precise cutoff.

Key ideas

• The mean can be updated from its prior estimate and the latest observation without storing the full sample.
• A recursive variance update can track dispersion alongside the mean.
• Exponential weighting gives newer data greater influence and supports constant-state streaming calculations.
• Matching centroids gives an approximate conversion between an SMA window and an EMA factor.
• When sampling frequency changes, the decay factor can be adjusted to preserve similar smoothing.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.