PCA: Eigenvectors, Scores, and Explained Variance

# What is PCA and how does it relate to eigenvectors and eigenvalues?


# What is PCA and how does it relate to eigenvectors and eigenvalues?












What are the principal components? How they are calculated? What is their relationship with eigenvalues and eigenvectors? This is a lead-in question to explain PCA basics.

EDIT: PCA is implemented in many software packages. It has lots of parameters which cause problems if one does not have a clear idea of what to expect as output and which parameters one should use. This lead-in question gives you some test cases to try, double check your outputs, and other interesting relationships between eigenvectors and eigenvalues.

## Answer by user12348 (score 8)

https://quant.stackexchange.com/a/11389

PCA(Principal Component Analysis) is the most interesting topic in QF. PCA is at the heart of quantitative data analysis. It is used in factor analysis, factor loadings, finding principal component of interest rate term structure for derivative and option pricing, data compression, eigenfaces( find the best match from a set of pictures with a , say, fuzzy face), you name it. So it behooves one to understand the basics because it will help one to understand the more advance concepts in PCA calculations.

I hope the answer and the thinking behind the logic is appealing and matches yours.

There are many ways to do this. I am using SVD, Eigen decomposition, which will work with matrix A being positive semi-definite. RMT is a relatively new field. To cover the basics this is a good start. Normally eigenvectors and eigenvalues relationship is shown as $$ A v = \lambda v $$ where $ v \space is \space eigenvector$ and $\lambda \space is \space eigenvalue $

An interesting property of eigenvector is that is you multiply it with a scalar, and multiply with A on the left, you will find vector reproduce on the right side, with factor k, with same eigenvalues. ( Exercise).

Mind you, it is a mistake to think that eigenvectors are same as principal components, however, they are related as shown below.

If you have a random dataset X as below, then what is principal component and how is data related to A matrix? Numerical example shows using MATLAB.

```
X = [269.8 38.9 50.5
272.4 39.5 50.0
270.0 38.9 50.5
272.0 39.3 50.2
269.8 38.9 50.5
269.8 38.9 50.5
268.2 38.6 50.2
268.2 38.6 50.8
267.0 38.2 51.1
267.8 38.4 51.0
273.6 39.6 50.0
271.2 39.1 50.4
269.8 38.9 50.5
270.0 38.9 50.5
270.0 38.9 50.5
];

[n m]=size(A)

n =
    15
m =
     3
```

Let us convert X to zscores, we can B=zscore(X) or for clarity B = (X - repmat(mean(X),[n 1])) ./ repmat(std(X),[n 1])

```
B =
   -0.0971   -0.0178    0.0636
    1.3591    1.5820   -1.5266
    0.0149   -0.0178    0.0636
    1.1351    1.0487   -0.8905
   -0.0971   -0.0178    0.0636
   -0.0971   -0.0178    0.0636
   -0.9932   -0.8177   -0.8905
   -0.9932   -0.8177    1.0178
   -1.6653   -1.8842    1.9719
   -1.2173   -1.3509    1.6539
    2.0312    1.8486   -1.5266
    0.6870    0.5155   -0.2544
   -0.0971   -0.0178    0.0636
    0.0149   -0.0178    0.0636
    0.0149   -0.0178    0.0636

>    cov(B)

ans =

    1.0000    0.9902   -0.8433
    0.9902    1.0000   -0.8831
   -0.8433   -0.8831    1.0000

>    corr(B)

ans =

    1.0000    0.9902   -0.8433
    0.9902    1.0000   -0.8831
   -0.8433   -0.8831    1.0000

>    corr(X)

ans =

    1.0000    0.9902   -0.8433
    0.9902    1.0000   -0.8831
   -0.8433   -0.8831    1.0000
```

Interesting Note: for standardized data mean=0, variance =1, $$A=cov(B)=corr(B)=corr(X)$$ However, note $$ cov(B) <> cov(X) $$ Therefore be careful of how you calculate the covariance matrix.

Note here A used in beginning is the covariance matrix of standardized input.

Decomposing matrix A or cov(B) gives us eigenvectors V and eigenvalues D as below.

```
[V D]=eig(cov(B))

V =

    0.6505    0.4874   -0.5825
   -0.7507    0.2963   -0.5904
   -0.1152    0.8213    0.5587

D =

    0.0066         0         0
         0    0.1809         0
         0         0    2.8125
```

Interesting to note that B'*B gives you a matrix proportional to cov(B).

> B'*B

```
ans =

   14.0000   13.8622  -11.8068
   13.8622   14.0000  -12.3640
  -11.8068  -12.3640   14.0000
```

It is off by a factor of 14.

> B'*B/14

```
ans =

    1.0000    0.9902   -0.8433
    0.9902    1.0000   -0.8831
   -0.8433   -0.8831    1.0000
```

Dividing by 14 gives you the covariance matrix , A or cov(B). Why is this so. Because Covariance estimator is given as: $$ \frac{1}{n-1} \sum_{i=1}^{n} (X_i- \overline{X}) {(X_i- \overline{X})}'$$ where $ \overline{X}=0 $ we get $$ Covariance matrix (estimator)=\frac{1}{n-1} B B' $$ That mystery is solved.

Note we can get matrix A or cov(B), also using Spectral Decomposition

> V*D*V'

```
ans =

    1.0000    0.9902   -0.8433
    0.9902    1.0000   -0.8831
   -0.8433   -0.8831    1.0000
```

Principal components are defined as scores in Matlab. We can get the principal components from eigenvalue

```
>    score=B*V

score =

   -0.0571   -0.0003    0.1026
   -0.1277   -0.1226   -2.5786
    0.0157    0.0543    0.0373
    0.0536    0.1326   -1.7779
   -0.0571   -0.0003    0.1026
   -0.0571   -0.0003    0.1026
    0.0704   -1.4579    0.5637
   -0.1495    0.1095    1.6299
    0.1041    0.2496    3.1841
    0.0319    0.3647    2.4306
    0.1093    0.2840   -3.1275
    0.0892    0.2787   -0.8467
   -0.0571   -0.0003    0.1026
    0.0157    0.0543    0.0373
    0.0157    0.0543    0.0373
```

We have three principal components above. Let us name them from left to right as PC1, PC2 and PC3. If you look at the eigenvectors last column you will see

```
[-0.5825        -0.5904         0.5587]'
```

The way the score is obtained from matrix multiplication between B and V, you can see that PC3 consists of -58.25% of first factor (in B) , -59.04% of 2nd factor and 55.87% of 3rd factor.

Which one is the most dominant?

```
>    var(score)

ans =

    0.0066    0.1809    2.8125
```

The right most vector, PC3, explains the most of the variance in the data X. you can do

> varB=sort(var(score),'descend') varB = ` 2.8125 0.1809 0.0066 ` cumsum(varB) / sum(varB) `ans = 0.9375 0.9978 1.0000 `

```
   2.8125    0.1809    0.0066
```

```
ans =

0.9375 0.9978 1.0000
```

So the dominant principal component(PC3) explains 94% of the variance. If you are finding the principal component that explains the variance in data then PCA works fine. If you are using PCA for data that has other information content then you need something different. Also, covariance matrix can be skewed by the outliers, I find that interesting. What if I am only interested in outliers. Outliers could be representing the Black Swans?! Then may be you are interested in the least important PC. You can plot the scores and see how it matches the outlier data.

One last thing, how do you get back the original data from score or principal components? As I showed earlier

```
>    B = (X - repmat(mean(X),[n 1])) ./ repmat(std(X),[n 1])
```

so

```
>    X = (B  .* repmat(std(X),[n 1]) + repmat(mean(X),[n 1]))
```

Which really is how we get X from B. How about the score? We know score is B*V so B=score*V' because then B=B*V*V' and V*V'=1 or identity matrix. therefore,

```
>    X = (score*V'  .* repmat(std(X),[n 1]) + repmat(mean(X),[n 1]))
```

In pseudo equation form, PC: principal component $$ original \space data = PC * eigenvector' * stdev(X) + mean $$

I have answered the following question in this

- how to get eigenvectors, eigenvalues and PCA

- how to relate eigenvectors to PCA components

- determine which eigenvalues explain the most variance

- how much do the factors contribute to the principal component

- double check which principal component is dominant

- How to get the original data back