Diagnosing Negative Eigenvalues in a Covariance Matrix
Summary
This note describes a reported mismatch between a covariance matrix that appears positive definite under Cholesky decomposition and negative eigenvalues returned by a symmetric eigensolver. The matrix is built from historical forward-rate data imported through pandas, with a covariance calculation and scaling applied. The proposed fix is to convert the dataframe to a NumPy array before calculating the covariance matrix.
The evidence is a single user report: after making that conversion, the author says the eigenvalues become correct. The note does not explain the underlying cause, provide a reproducible example, or establish whether dataframe structure, data types, or another detail caused the discrepancy. It is therefore a troubleshooting anecdote rather than a general numerical analysis of covariance matrices; checking symmetry, input values, and numerical precision may still be necessary in other cases.
Key ideas
- A symmetric covariance matrix should have nonnegative eigenvalues when it is positive semidefinite.
- The reported issue involves covariance data imported and processed as a pandas dataframe.
- Converting the dataframe to a NumPy array before covariance calculation reportedly resolves the discrepancy.
- The note offers no diagnosis of why the conversion changes the result, so the fix may not generalize.
Tags
Full text
# Why are my eigenvalues coming out negative for my positive definite covariance matrix?
# Why are my eigenvalues coming out negative for my positive definite covariance matrix?
I have a 51 x 51 covariance matrix that is derived from historic forward rates that is positive definite. I know it is because in Python np.cholesky returns a correct cholesky decomposition. However, when I use the eigh function to return the eigenvalues/vectors, about half of the eigenvalues are negative, which makes no sense. The data is imported from excel using pandas dataframes, then I make a covariance matrix using:
```
CovMat2 = np.cov(df.T)*252/10000
```
Then I request the eigenvalues with
```
eig_vals, eig_vecs = np.linalg.eigh(CovMat2)
print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)
and I get this output:
Eigenvalues
[-1.35963506e-03 -1.09487110e-03 -5.00507744e-04 -4.76180253e-04
-4.26481890e-04 -3.65765795e-04 -3.21245207e-04 -2.90812021e-04
-2.49174935e-04 -2.47519558e-04 -2.20946393e-04 -2.11019230e-04
-2.01940366e-04 -1.99581473e-04 -1.87201389e-04 -1.74938799e-04
-1.41441186e-04 -1.24247169e-04 -1.11167101e-04 -9.78191552e-05
-8.75246993e-05 -8.69367038e-05 -8.36134890e-05 -8.03187245e-05
-7.36730798e-05 -5.70815773e-05 -4.65981850e-05 -2.92229574e-05
-2.77958421e-05 -1.84175309e-05 -7.45464337e-06 1.37165843e-05
2.91872261e-05 3.93959482e-05 4.46749803e-05 6.07783927e-05
7.42504887e-05 7.99764211e-05 9.49821563e-05 1.05894570e-04
1.15379642e-04 1.37420008e-04 1.47812090e-04 2.07834453e-04
2.63958555e-04 2.75403037e-04 3.01136833e-04 3.06428511e-04
3.75549199e-04 1.96296188e-03 5.80951459e-03]
```
which makes no sense. What am I doing wrong? Perhaps it's something to do with converting dataframes to a numpy matrix?
## Answer by M Thomas (score 1)
https://quant.stackexchange.com/a/40677
It appears that changing the dataframe to values sorts this problem out. df = df.values before calculating the covariance matrix gives correct eigenvalues.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.