Why NumPy and Pandas Covariance Calls Produce Different Matrices
Summary
The document compares covariance estimates from a pandas returns table and a NumPy array, where the results differ in shape and contain missing values. The core issue is that the two interfaces interpret the data layout differently: pandas computes covariances between columns, while NumPy’s covariance routine treats rows as variables by default. Passing a time-by-asset table without adjusting that orientation therefore yields covariances across the wrong dimension.
The displayed pandas result is a six-by-six asset covariance matrix, while the NumPy output is based on a different orientation and includes a column affected by missing returns. To reproduce the intended asset matrix, the data must be oriented so assets are treated as variables and missing observations must be handled consistently. The example illustrates an implementation pitfall rather than a statistical comparison; it does not discuss estimation windows, annualization, or other covariance-model choices.
Key ideas
- Pandas covariance on a DataFrame calculates pairwise covariance across columns.
- NumPy covariance treats each row as a variable unless its orientation option is changed.
- A time-by-asset return array can therefore produce a matrix with the wrong dimension and interpretation.
- Missing return observations can propagate into covariance estimates and require consistent handling.
Tags
Full text
# What is the difference between np.cov(array) and array.cov()?
# What is the difference between np.cov(array) and array.cov()?
I'm trying to find a covariance matrix, so when i use `returns.cov()` on my returns variable, I get a good result. Unfortunately, when i want to use `.cov()` on my np array converted from pandas df, i must use np.cov(array). But then, i get a completely different matrix, that is not even the dimensions that it should be. How to get a normal covariance matrix while using np.cov(array) or how to convert this crazy matrix into a normal one? Here is my code:
```
returns = np.log(data).diff()
cov_matrix = returns.cov()
cov_matrix
```
Output:
```
GE MSFT JNJ K BA IBM
GE 0.103503 0.075849 0.022159 -0.004298 0.085745 0.012300
MSFT 0.075849 0.165744 0.041228 -0.005173 0.037925 -0.002240
JNJ 0.022159 0.041228 0.035968 0.018111 0.010101 -0.003933
K -0.004298 -0.005173 0.018111 0.056953 -0.007621 -0.004619
BA 0.085745 0.037925 0.010101 -0.007621 0.089571 0.024823
IBM 0.012300 -0.002240 -0.003933 -0.004619 0.024823 0.018391
```
here is the other code using np.cov(array):
```
cov_matrix = np.cov(returns)
cov_matrix
```
output:
```
array([[ nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan,
nan, nan],
[ nan, 0.08709179, -0.02110741, 0.02440574, 0.028228 ,
0.01615291, -0.00658686, 0.07601494, 0.03515373, -0.04674915,
-0.03947769, 0.03605138],
[ nan, -0.02110741, 0.02745833, 0.00419789, 0.00786532,
0.00018428, 0.03678159, -0.00574526, -0.02711512, 0.01241536,
-0.00982581, -0.01954344],
[ nan, 0.02440574, 0.00419789, 0.02056029, 0.0081328 ,
0.01348074, 0.0125812 , 0.02001429, 0.00479978, -0.01069634,
-0.02331694, 0.00175948],
[ nan, 0.028228 , 0.00786532, 0.0081328 , 0.08130515,
0.01101311, 0.07589644, 0.06730002, -0.04176551, -0.02758789,
-0.04104097, 0.01693321],
[ nan, 0.01615291, 0.00018428, 0.01348074, 0.01101311,
0.02619104, 0.0244941 , 0.0028883 , -0.01552481, -0.01530776,
-0.0314189 , 0.00692483],
[ nan, -0.00658686, 0.03678159, 0.0125812 , 0.07589644,
0.0244941 , 0.11670208, 0.03269149, -0.08127712, -0.01149054,
-0.05617884, -0.00707836],
[ nan, 0.07601494, -0.00574526, 0.02001429, 0.06730002,
0.0028883 , 0.03269149, 0.10320532, 0.01023065, -0.04249818,
-0.04005964, 0.03031267],
[ nan, 0.03515373, -0.02711512, 0.00479978, -0.04176551,
-0.01552481, -0.08127712, 0.01023065, 0.0724934 , -0.0045111 ,
0.02626014, 0.01213532],
[ nan, -0.04674915, 0.01241536, -0.01069634, -0.02758789,
-0.01530776, -0.01149054, -0.04249818, -0.0045111 , 0.03077156,
0.02965359, -0.02410751],
[ nan, -0.03947769, -0.00982581, -0.02331694, -0.04104097,
-0.0314189 , -0.05617884, -0.04005964, 0.02626014, 0.02965359,
0.05648698, -0.01213576],
[ nan, 0.03605138, -0.01954344, 0.00175948, 0.01693321,
0.00692483, -0.00707836, 0.03031267, 0.01213532, -0.02410751,
-0.01213576, 0.02393791]])
```Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.