Diagnosing Non-Positive-Definite Covariance Matrices in PCA
Summary
The document discusses why a covariance matrix of changes in forward rates may fail to be positive definite when used for principal component analysis. It distinguishes a matrix with small negative eigenvalues caused by floating-point round-off from a matrix that is singular because the observations contain linearly dependent variables. The responses suggest checking the rank of the original data matrix and looking for redundant series, including through unusually high pairwise or partial correlations and similar variable scores across principal components.
The discussion is practical but limited: the data are not supplied, so the specific cause cannot be diagnosed. The suggested correlation and component-score checks are described as brute-force methods that may not scale. If the covariance matrix was calculated conventionally from complete data, materially negative eigenvalues would not be expected; tiny numerical negatives may be treated as zero, while deficient rank calls for investigating the input series.
Key ideas
- A covariance matrix can be positive semidefinite without being positive definite when variables are linearly dependent.
- Near-zero negative eigenvalues may arise from numerical round-off in an otherwise valid covariance estimate.
- Checking the rank of the input data can reveal whether the covariance matrix is singular.
- High pairwise or partial correlations and similar component scores can help locate redundant variables.
- Without the underlying observations, the matrix's specific failure mode cannot be determined.
Tags
Full text
# Why is my Covariance matrix not positive definite? # Why is my Covariance matrix not positive definite? I'm trying to do PCA on historic forward rates. I'm using forward rates from the Bank of England going from Jan 2015 through end of May 2018. I calculate the differences in the rates from one day to the next and make a covariance matrix from these difference. The matrix is 51 x 51 (because the tenors are every 6 months to 25 years plus a 1 month tenor at the beginning). My matrix is not positive definite which is a problem for PCA. I don't understand why it wouldn't be. The data is "clean" (no gaps). ## Answer by DJohnson (score 1) https://quant.stackexchange.com/a/41090 'Not positive definite' is an algebraic statement that some of the variables are linear combinations of one another. The problem then becomes one of tracking down the offending variates. I've used two brute-force approaches for this but neither scales well in the presence of large amounts of information. One method is to examine pairwise correlations and partial correlations looking for very high r-values, e.g., r>=0.95. A second tactic is much more nitty-gritty and involves scrutinizing the variable-level scores across the resulting components as output from the PCA. By sorting the variables on their first few components one can identify variables with the same or highly similar score values. I'm sure other QF participants have much more sophisticated tactics that do scale well to large data. ## Answer by Enrico Schumann (score 0) https://quant.stackexchange.com/a/40613 You have not shown data, so one can only guess. If you have computed the covariance matrix from the full dataset with no missing values (and you have not used some weird estimator), then the only way to have negative eigenvalues is round-off error: in that case, those negative eigenvalues will be practically zero: so just replace them with zero. See http://comisef.wikidot.com/tutorial:repairingcorrelation. A different question is whether your covariance matrix has full rank (i.e. is definite, not just semidefinite). If you have at least `n+1` observations, then the covariance matrix will inherit the rank of your original data matrix (mathematically, at least; numerically, the rank of the covariance matrix may be reduced because of round-off error). So you should check your original data matrix, whether it has rank 51, or less.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.