Diagnosing Singular Covariance Matrices and Shrinkage Failures
Summary
The document examines why a covariance matrix for a large stock universe may remain singular after applying Ledoit–Wolf shrinkage. It suggests checking whether securities are duplicates or nearly perfectly correlated, since multicollinearity can make the underlying covariance matrix rank deficient. The discussion also notes that monthly data with more assets than observations cannot support an invertible sample covariance matrix.
Possible remedies and failure checks include alternative shrinkage estimators, dynamic conditional correlation models, exponentially weighted covariance estimates, robust covariance methods, and bootstrap intervals for assessing uncertainty. The answers emphasize that the theoretical positive-definiteness guarantee depends on the estimator's assumptions: a positive-definite shrinkage target and a positive-semidefinite sample covariance. Missing observations handled pairwise can violate that condition, and software bugs or implementation differences may also explain the result. These approaches are suggestions rather than a comparative evaluation; the document provides no empirical test establishing which method works best for a given portfolio.
Key ideas
- Perfect or near-perfect correlations among assets can make a covariance matrix singular.
- When asset count exceeds the number of observations, the sample covariance matrix cannot be full rank.
- Ledoit–Wolf positive-definiteness depends on valid inputs and a positive-definite shrinkage target.
- Pairwise handling of missing data can produce a sample covariance matrix that is not positive semidefinite.
- Alternative estimators and implementations should be checked, but the document does not compare their performance.
Tags
Full text
# Ledoit-Wolf Shrinkage estimator not giving positive definite covariance matrix # Ledoit-Wolf Shrinkage estimator not giving positive definite covariance matrix I used ten year daily data for 407 stocks and computed the daily and monthly covariance matrices. Since I have more variables than observations for the monthly matrix, I wasn't surprised to find the matrix to be not invertible (and hence useless for portfolio optimization). I was surprised to see the daily covariance matrix not invertible. I then tried to shrink the matrix with the Ledoit-Wolf shrinkage estimator using the package tawny. It didn't help. It makes the covariance matrix really, really small, but no invertible. Does anyone have any suggestions what could be the problem? How could I improve the covariance matrix? ## Answer by Richi Wa (score 2, accepted) https://quant.stackexchange.com/a/19521 Are your 407 stocks all different? No A and B listings contained that are strongly if not perfectly correlated? The observation that the daily covariance matrix is singular makes me wonder. You can try the package corpcor for another shrinkage estimator. ## Answer by JPN (score 3) https://quant.stackexchange.com/a/19611 The problem with Ledoit-Wolf is that it's very sensitive to outliers. You should try these: - DCC GARCH unfortunately, not available in Python - Exponentially weighed moving average (EWMA) gives slighly worse results than DCC-GARCH - Minimum Covariance Determinant suggestted by Scikit-Learn - bootstrap could be used to calculate confidence interval, then you can decided how conservative you want to be Here are some good references: - The Impact of Covariance Misspecification in Risk-Based Portfolios - A Test of Covariance Matrix Forecasting Methods - Scikit Learn Covariance Manual ## Answer by Matifou (score 3) https://quant.stackexchange.com/a/49805 In theory, the Ledoit and Wolf shrinkage estimator is supposed to guarantee a positive-definite matrix, given that it adds a positive-definite matrix (the target) to a semi-positive one (the sample covariance). I can see four reasons why you didn't get a positive-definite matrix: - Your true covariance is effectively not full rank, i..e you have perfect multicolinearity - Your target is not positive definite? That is something you can check easily. Taking (a multiple of) the identity will however guarantee it is - Your sample covariance matrix is not semi-positive definite: this can happen when you had missing values, and used a `cor(x, "pairwise.complete.obs")` approach - There are bugs in the code (`tawny` has indeed bugs in the Ledoit Wolf estimator, as of version 2.1.7). Check alternatives like `nlshrink::linshrink_cov()` and `CovTools::CovEst.2003LW()`
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.