How Variance and Expected Shortfall Differ in Portfolio Risk Optimization

# Minimizing variance vs. expected shortfall: distributions where the difference is salient


# Minimizing variance vs. expected shortfall: distributions where the difference is salient












In portfolio theory in finance, given a set of $n$ assets to choose from, one often selects portfolio weights so as to maximize expected return and minimize some measure of risk, e.g. variance or expected shortfall*. If we consider asset returns to be random variables, we are looking for a linear combination, with weights summing up to unity, of random variables that has a high expected value and a low measure of risk. Let us consider just two assets, $n=2$, and suppose expected returns of all assets are equal to the same constant, e.g. zero.

Questions:

- What are some bivariate distributions that allow minimizing variance but retaining large expected shortfall, or vice versa?

- What are some bivariate distributions that would produce substantially different optimal weights when minimizing variance vs. minimizing expected shortfall?

- What is characteristic to such distributions? (Realistic examples from finance would be appreciated.)

*$q\%$ expected shortfall (a.k.a. expected tail loss or conditional value at risk) is simply the mean of the left tail of the random variable, the tail being cut off at the $q\%$ quantile level.

The topic covers both finance and statistics. The question has been previously posted at Cross Validated but received zero answers. I am now reposting it here. While cross posting is not recommended, I will keep the post on Cross Validated in case it might eventually attract an answer there.

## Answer by g g (score 1)

https://quant.stackexchange.com/a/57851

Such calculations quickly get messy even in the bivariate case and are best addressed by simulations. That said, the basic question about the fundamental difference between optimisation using Tail risk versus Variance based risk measures can be illustrated by a straightforward calculation using only the total portfolio return.

Put simply the philosophical and practical difference is that Tail risk measures only focus on the tails while Variance incorporates information from the whole distribution. All other differences then follow from this basic distinction.

### Tail/Non-Tail Decomposition

I think it is completely sufficient to analyse the univariate case. Let $S$ denote the total portfolio return (e.g. $S = wX + (1-w)Y$ for two assets $X$ and $Y$ with weight $0\leq w \leq 1$).

With the tail probability $0<q < 1$ and the tail quantile $s_q$ ( i.e. $\mathbb{P}[S<s_q] = q$) we can distinguish between the tail $\{ S \leq s_q\}$ and non-tail $\{ S > s_q\}$ regions of $S$ using the Bernoulli Variable $Z = \mathbb{1}_{\{ S \leq s_q\}} $. Let $F_S$ be the distribution of $S$ and $\hat{F} = F_S \mid \{Z = 0\}$ be the upper or non-tail conditional distribution and $\check{F} = F_S \mid \{Z = 1\}$ be the lower, tail conditional distribution. Those distributions are lower respectively upper truncated distributions. Furthermore, we need $\hat{e}$ and $\check{e}$ the expectations as well as the variances $\hat{v}^2$ and $\check{v}^2$ of $\hat{F}$ and $\check{F}$.

For simplicity assume that $S$ has a continuous density. Then $-\check{e}$ is the expected shortfall of $S$. By the law of total expectation using $\mathbb{E}[S]=0$ one sees that: $$ 0 = \mathbb{E}[S] = q \check{e} + (1 - q)\hat{e}$$ or $$\hat{e} = -\frac{q}{1-q}\check{e}.\tag{1}\label{1}$$

In the same way, only now using the law of total variance, we can take apart the Variance of $S$: $$ \begin{align}\mathbb{V}ar[S] &= \mathbb{E}[\mathbb{V}ar[S\mid Z]] + \mathbb{V}ar[\mathbb{E}[S\mid Z]\\ &= q \check{v}^2 + (1 - q)\hat{v}^2 + \frac{q}{1-q}\check{e}^2\tag{2}\label{2}. \end{align}$$ For the third term one uses the fact that $Z$ is Bernoulli with $\mathbb{P}[Z=1]=q$ and the relation $(\ref{1})$ between the two possible values of $\mathbb{E}[S\mid Z].$

### Interpretation

According to $(\ref{2})$ the variance can be decomposed into two "within" variances i.e. tail and non-tail variance and an "in-between" variance arising from the difference in mean between tail and non-tail.

So yes indeed, a large expected shortfall will drive the variance. In that sense optimisation of variance and expected shortfall will provide one with similar directions. But the variance incorporates additional terms, which are completely ignored by expected shortfall optimisation. And while arguably and in practice often $\check{v}^2$ will be closely related to $\check{e}$ by the tails of the available asset distributions, the behaviour of $\hat{v}^2$ is often quite separate and somewhat dominant, especially if $q$ is very small. Under Variance optimisation it makes a lot of sense to take some more tail risk to get rid of non-tail volatility.

This myopic behaviour is also the reason why pure expected shortfall (or value at risk) optimisation will be rare in practice. It is no consolation to be well-managed on a 1-in-100 years level, if you regularly incur losses.