Handling Negative Values in the Zhou High-Frequency Volatility Estimator

# Negative high frequency intraday volatility - Zhou estimator


# Negative high frequency intraday volatility - Zhou estimator












To estimate high frequency tick data stock intraday volatility, I have read Robert Almgren's notes7.pdf

http://www.cims.nyu.edu/~almgren/timeseries/notes7.pdf

where he talks about the bias free estimator by Zhou:

$Z = \sum ((y_j - y_{j-1})^2 + 2(y_j - y_{j-1})(y_{j+1} - y_j))$

where $y_j$ is the log return of the price at time $j$.

However, this expression sometimes yields negative volatility. We see that the first term is a square which is always positive, but the second term $2(y_j-y_{j-1})(y_{j+1}-y_j)$ can be negative. So how do I treat this estimator? I want a positive volatility, not a negative! I am thinking of just applying absolute value on the second term, but that does not sound right. Any suggestions?

## Answer by Colin T Bowers (score 8, accepted)

https://quant.stackexchange.com/a/18969

In any finite sample, it is always possible for the Zhou estimator to return a negative number, even though we know the unobservable parameter being estimated is non-negative. This is a well known issue in the academic literature. There are several approaches to dealing with this problem:

1) Ignore it. (I don't like this one). It is particularly nefarious if you ever move to a multivariate setting and you suddenly end up with a covariance matrix that is not positive semi-definite. (try taking the inverse of it and watch everything go to hell in your code).

2) Set your final estimator equal to $max(v, 0)$, where $v$ is the Zhou estimator. In practice, there is nothing wrong with this ad hoc estimator, although a pure theorist might get cross about the fact that the asymptotic distribution is affected. From a coding perspective, you might even want to use $max(v, \alpha)$ for some small positive constant $\alpha$ so you don't accidentally end up dividing by zero anywhere. NOTE: please do not take the absolute value of the autocovariance portion of the estimator, as you mention in the question. This makes no sense from an estimation perspective and will result in a heavily biased estimator that is inconsistent even under ideal modelling assumptions.

3) Use a different intraday-data based estimator of volatility that doesn't suffer from this problem. I strongly recommend this option. The Zhou estimator was state-of-the-art in 1996 (and that paper itself was astonishingly pre-scient - it took another decade for everyone else to catch on to the problems Zhou tried to solve in that paper), but a lot of work since then has demonstrated that it will be heavily biased for many high-frequency datasets, see e.g. Hansen, Lunde (2006) "Realised Variance and Market Microstructure Noise". Probably the most popular estimator at the present point in time is the realised kernels estimator of Barndorff-Nielsen, Hansen, Lunde, and Shephard. The original 2008 paper is a bit heavy going for a non-theorist, but have a look at Barndorff-Nielsen, Hansen, Lunde, and Shephard (2009) "Realized Kernels in Practice" - it is much more friendly. (I have an implementation of this estimator in the Julia language based off this paper).

This estimator also has the nice property that if you use a Parzen kernel (see the above reference for more detail), then the estimator will always be non-negative. Be warned, for other types of kernel function, especially flat-top kernels, there is no guarantee of non-negativity. See eg footnote 2 of the above-mentioned paper.

If you want something simpler in that you can implement quickly, then just use a low sampling frequency realized variance, e.g. 5-minute realized variance or maybe 10-minute. Since this is just a sum of squared intrady returns, it is guaranteed to always be non-negative and will, in my experience, provide a better estimate than the Zhou estimator anyway.

## Answer by Will Gu (score 1)

https://quant.stackexchange.com/a/51137

If you calculate this estimator point by point, it could very well go negative. But the essence of this estimator is that statistically, it hovers around the realized variance, so it "estimates" the realized variance well. Even though the realized variance is by definition non-negative, the estimated distribution does not have to be.

In practice can you really use this estimator in place of real-time variance? It's like shooting a target through a blurred lens. You could miss by a lot.

A potential use case of this estimator though, is that if you only care about high variance moments in your trading algorithm. Negative values in this case become irrelevant. Zhou's estimator is easily calculated online as market data streams in.

I found this presentation by Jim Gatheral in 2006 on this subject. His conclusion is that Zhou's estimator outshined other estimators in the survey.

By the way I learned something from other top answers, especially the realized kernel method. It looks to me that Zhou's original paper set up the ground, and others generalized the problem into a kernel problem. Now it just becomes a study of different kernels.

## Answer by LazyCat (score 0)

https://quant.stackexchange.com/a/17928

I think it's ok: the total correction is $\approx -N\eta^2 \sim N^{-1/2},$ which tends to zero for large $N.$ So as long as it is a correction to the estimator $Q > 0$ you are ok.

## Answer by user16313 (score 0)

https://quant.stackexchange.com/a/17941

Well for real hf stock tick data, i do get negativa volatility. So it can happens for large N