Why Lower Weight Error Can Still Produce Higher Portfolio Variance

Summary

The document describes a simulation designed to compare covariance transformations for minimum-variance portfolio construction. For each lookback window, the author samples portfolios of 100 assets, estimates a sample covariance matrix, transforms it, and compares the resulting optimized weights with weights based on covariance observed during the 21-day holding period. Repeated trials are used to compare weight RMSE and out-of-sample variance under long-only, fully invested constraints.

The reported puzzle is that non-linear shrinkage yields lower weight RMSE across the tested lookback periods but higher out-of-sample portfolio variance. The document provides the experimental setup and observation, but no explanation or underlying numerical results. It highlights that closeness of portfolio weights to a reference solution does not necessarily imply better realized risk; the conclusion is limited to the author’s described simulation and its chosen asset samples, constraints, and evaluation period.

Key ideas

• The experiment compares transformed sample covariance estimates with covariance observed over the portfolio holding period.
• It evaluates optimized portfolios using both weight RMSE and out-of-sample variance.
• The author reports lower weight RMSE but higher realized variance for portfolios using non-linear shrinkage.
• The portfolios are long-only and use the full available leverage budget.
• The document reports a puzzle rather than a causal explanation, and its finding is specific to the described simulation.

Tags

# Intuition behind portfolio weights with lower RMSE but higher variance


# Intuition behind portfolio weights with lower RMSE but higher variance












I have recently encountered a phenomena in portfolio optimization that has baffled me for days. I was experimenting with different ways of transforming a covariance matrix to get a stable minimum variance portfolio out-of-sample. To do this, I employed this methodology:

- Set a portfolio holding period of 21 days.

- Determine range of lookback windows, where the minimum lookback window is 21 days.

- For each lookback window, randomly sample a portfolio of 100 assets.

- Calculate both the sample covariance matrix and the "oracle" covariance matrix (covariance matrix in the holding period).

- Apply transforming method to the sample covariance matrix to get the transformed covariance matrix.

- Feed the oracle and transformed covariance matrix into a minimum variance portfolio optimizer to obtain weights.

- Repeat 1000 times for each lookback window.

- Verify that the "oracle" weights indeed produce the minimum variance portfolio on average, and benchmark weights produced by the different transformation schemes against the "oracle" weights by calculating their RMSE.

Intuitively, a lower RMSE should lead to a lower out-of-sample variance (in the holding period). But I have found that not to be true for non-linear shrinkage. Weirdly enough, non-linear shrinkage consistently produces portfolio weights with lower RMSE (across all lookback periods) but higher variance out-of-sample. Not sure why that is the case, and I'm hoping you guys can give some insights!

Portfolio constraints were:

- long-only

- leverage set at 1

Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.