When Standard Deviation Misleads: Bias, Heavy Tails, and Missing Variance

# How can certain numerical distributions yield misleading standard deviation calculations?


# How can certain numerical distributions yield misleading standard deviation calculations?












What numerical distributions yield misleading standard deviation calculations? Can you make the standard deviation of distribution 1 attain a higher value than the standard deviation of distribution 2 where they are from two different discrete probability distributions?

I am aware that standard deviation is sensitive to the number of sample points.

Also, I am interested in the same question for downside standard deviation.

Thank you for reading this

## Answer by Dave Harris (score 1)

https://quant.stackexchange.com/a/31404

There are no distributions that have a standard deviation that trigger misleading calculations. Not all distributions have a standard deviation. The standard textbook estimator for variance is an unbiased estimator for variance, but it is a biased estimator for the standard deviation. This is because taking the square root of an expectation, which is what you are doing when you take the square root of the estimate of sample variance, is not necessarily equal to taking the expectation of the square root, which is what an unbiased estimator would do. This is known as Jensen's inequality.

Because of this you can sort of play small sample size games on purpose with a handful of discrete distributions that have been rigged to trigger this property, but that is more of a parlor trick.

Now among the continuous distributions, the distribution of returns for most assets lacks the property normally called variance. The US stock market returns for going concerns is close to a truncated Cauchy distribution, with the slight skew being due to the budget constraint.

A standard deviation can be thought of as a property, like a nose is a property of many animals, but absent in trees. If you saw something with a nose, you would not say it was a tree. Some distributions have no standard deviation and so any calculation of a standard deviation will be misleading as it will generate a random number instead. The symmetric Cauchy distribution has the peculiar property that as the sample size goes to infinity, the sample variance will go to infinity. Conversely, when you do this with a normal distribution you will get a near perfect estimate of the true variance.

Nonetheless, the only way to trigger this type of behavior in discrete distributions is to bin a distribution that has no variance. Binning will not cause a variance to appear, but it would be a weird thing to do with a discrete distribution, on purpose.

The symmetric Cauchy distribution is $$\frac{1}{\pi}\frac{\sigma}{\sigma^2+(x-\mu)^2}.$$ With a little bit of creativity and a good proper understanding of how to solve the constant of integration and evenly spaced bins, you could create a discrete distribution, say over the values of the Cauchy distribution on the integers, that would generate a misleading variance because it would not have one.

You can find a derivation for the returns on various asset classes at https://ssrn.com/abstract=2828744 It is technical however so I wouldn't worry about it too much. It is listed more for completeness. As your technical skills increase, you can come back to it.