Using an EMA in Exponentially Weighted Standard Deviation Calculations
Summary
The document asks whether TA-Lib can speed up calculation of an exponentially weighted moving standard deviation. It distinguishes TA-Lib’s rolling standard deviation from the exponentially weighted statistic provided by pandas. The proposed custom Numba routine recomputes weights and weighted moments over the full history at every observation, which is inefficient on large datasets.
The answer says TA-Lib has no built-in exponentially weighted standard deviation, but suggests using its EMA function for the moving mean inside a custom calculation. The example retains a loop over observations and recomputes weighted deviations and a bias adjustment, so it does not demonstrate a fully incremental variance algorithm or establish a speed improvement. Its EMA initialization and matching of weighting conventions to pandas also require care before treating the result as equivalent.
Key ideas
- TA-Lib’s rolling standard deviation is not the same statistic as an exponentially weighted standard deviation.
- The document says TA-Lib has no built-in exponentially weighted standard deviation function.
- The suggested approach uses TA-Lib’s EMA as the moving mean in a custom weighted variance calculation.
- The example still recomputes weights and deviations for each observation, so it does not establish efficient scaling or exact pandas equivalence.
Tags
Full text
# How to use Ta-Lib to calculate the ewmstd?
# How to use Ta-Lib to calculate the ewmstd?
I am trying speed up below code for faster speed,
```
df = df.ewm(span=lookback).std()
```
I wrote a numba version but it is very slow in large dataset
```
from numba import njit
@njit
def ewmstd(arr, span):
'''
var_pandas = df.ewm(span=N).var()
std_pandas = df.ewm(span=N).std()
'''
std_arr = np.empty_like(arr)
N = span # Span
a = 2./(1+N) # Alpha
std_arr[0] = np.nan
for i in range(1, len(arr)):
z = arr[:i+1]
# Get weights: w
n = len(z)
w = (1-a)**np.arange(n-1, -1, -1) # This is reverse order to match Series order
# Calculate exponential moving average
ewma = np.sum(w * z) / np.sum(w)
# Calculate bias
bias = np.sum(w)**2 / (np.sum(w)**2 - np.sum(w**2))
# Calculate exponential moving variance with bias
ewmvar = bias * np.sum(w * (z - ewma)**2) / np.sum(w)
# Calculate standard deviation
ewmstd = np.sqrt(ewmvar)
std_arr[i] = ewmstd
return std_arr
```
Looking in TA-Lib, `STDDEV` is the rolling std dev, not ewmstd (the exponentially-weighted moving std). Could I couple some TA-Lib function to compose the ewmstd ?
## Answer by Hans-Peter Schrei (score 0)
https://quant.stackexchange.com/a/74928
As you say, TA-Lib does not have a built-in function to calculate the exponentially-weighted moving standard deviation. You can use ta.EMA in your function. Is this what you are looking for?
```
import numpy as np
import talib as ta
def talib_ewmstd(arr, span):
N = span
a = 2. / (1 + N)
# Calculate EMA using TA-Lib
ema = ta.EMA(arr, timeperiod=N-1)
# Create an empty array for EWMSTD values
ewmstd_arr = np.empty_like(arr)
ewmstd_arr[0] = np.nan
for i in range(1, len(arr)):
z = arr[:i + 1]
if i < N - 1:
ewmstd_arr[i] = np.nan
continue
# Get weights: w
n = len(z)
w = (1 - a) ** np.arange(n - 1, -1, -1) # This is reverse order to match Series order
# Calculate bias
bias = np.sum(w) ** 2 / (np.sum(w) ** 2 - np.sum(w ** 2))
# Calculate exponential moving variance with bias
ewmvar = bias * np.sum(w * (z - ema[i]) ** 2) / np.sum(w)
# Calculate standard deviation
ewmstd = np.sqrt(ewmvar)
ewmstd_arr[i] = ewmstd
return ewmstd_arr
```Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.