How Ledoit–Wolf Covariance Shrinkage Affects Asset Variances
Summary
The document clarifies why different Ledoit–Wolf covariance estimators can produce different diagonal values. Shrinkage is not one unique procedure: the result depends on the target matrix. The constant-variance approach used by scikit-learn shrinks toward a diagonal matrix whose entries are the average sample variance, so individual asset variances can move as well as covariances. A single-factor target may also alter variances.
By contrast, constant-correlation shrinkage keeps sample variances unchanged while pulling pairwise correlations toward their average, thereby changing covariances. The cited pedagogic paper follows this latter approach in its examples, which explains why its variances remain fixed. The practical lesson is to identify the estimator and target rather than assume every method bearing the Ledoit–Wolf name preserves the diagonal. The discussion is conceptual and does not compare estimator performance on a dataset.
Key ideas
- Ledoit–Wolf refers to multiple covariance shrinkage estimators with different targets.
- Constant-variance shrinkage pulls asset variances toward their average and can change diagonal entries.
- Constant-correlation shrinkage preserves sample variances while shrinking pairwise correlations.
- A paper that leaves variances unchanged may be using constant-correlation shrinkage.
- Check the target implemented by a library before interpreting changes in a covariance estimate.
Tags
Full text
# Covariance Shrinkage - Am I getting the right variances? # Covariance Shrinkage - Am I getting the right variances? I am looking into a quite simple task: shrinking the sample covariance matrix of a minor sample of monthly returns data on 5 different assets. I am using Python to process my data and have been using two different libraries (sklearn and PyPortfolioOpt - both using the Ledoit & Wolf method), which both yield the same problem: My diagonal values (variances) on my assets are way off from the sample variances. According to the practical paper "An Introduction to Shrinkage Estimation of the Covariance Matrix: A Pedagogic Illustration" (Kwan, 2011), my variances are not supposed to change at all during shrinkage - only the covariances. Can you guys confirm that only my covariances are supposed shrink and not my variances? If so, can any of you guys explain why I am getting skewed variances when using the Python libraries? Thanks in advance! ## Answer by nbbo2 (score 2, accepted) https://quant.stackexchange.com/a/63971 Ledoit-Wolf published several papers on shrinkage, so "Ledoit Wolf" does not uniquely identify the method. The PyPortfolio Opt documentation describes the available options as follows Ledoit-Wolf shrinkage: - constant_variance shrinkage, i.e the target is the diagonal matrix with the mean of asset variances on the diagonals and zeroes elsewhere. This is the shrinkage offered by sklearn.LedoitWolf - single_factor shrinkage. Based on Sharpe’s single-index model which effectively uses a stock’s beta to the market as a risk model. See Ledoit and Wolf 2001 [4]. - constant_correlation shrinkage, in which all pairwise correlations are set to the average correlation (sample variances are unchanged). See Ledoit and Wolf 2003 [3] Only the third choice, constant correlation shrinkage from the original Ledoit Wolf 2003 paper leaves the variances alone and only alters the covariances. The first method will reduce large variances and increase small variances. I believe the second method (single factor shrinkage) will also (if I recall correctly). The Kwan paper, on the other hand, takes the third approach, at least for the examples, and therefore does not change the variances.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.