Selecting Sparse Features to Represent Principal Components

# Answer by Attack68 (score 3)


# PCA: How to select a smaller set of the original features that best represent first PC with minimal contribution to the other PCs












I have performed PCA on a covariance matrix. I have 24 original features and subject to some constraints over which features are used i would like to chose the combination of features that best represent the first PC i.e. choose a smaller set of original features to represent PC1 with as small a contribution to the other components as possible.

The optimal solution is of course the 1st PC itself but say i want to limit to a subset of the original features, which should i pick and how to calculate the ratios.

This can be done approximately/visually by looking at the relative contribution of each feature to each PC and selecting only those that only have net contribution to the first PC and 0 to any other PC, but what other ways of doing this exist?

For example, from the below, you could approximately say that choosing features 1 and 24 will approximate PC1 without contributing to PC2, but it does have contribution to PC3... i was looking for a systematic way to make this choice given and a fixed list of features to chose from.

## Answer by Attack68 (score 3)

https://quant.stackexchange.com/a/45683

What you are describing is not mathematically plausible.

Firstly, but less important, a PC is a normalised vector (an eigenvector) meaning if it has more than one non-zero element they will always be less than one. Of course you can scale the PC but technically any feature will never be worth one in the direction of the PC unless every other feature has zero weight in that same PC.

Secondly, and importantly, PCA is a coordinate basis transformation, meaning if your original space is truly 24 dimensional (covariance matrix has rank 24), you will never return a PCA decomposition with what you require.

As an example suppose you get a PCA decomposition of the following 4 dimensional space which satisfies your requirements for features 1 and 2 (these are scaled, not orthonormal):

```
      PC1   PC2   PC3   PC4
f1    1     0     0     0
f2    1     0     0     0
f3    1     -1    -1    0
f4    1     1     -1    0
```

This does not define a 4 dimensional basis for your original instruments. What it tells you in fact is that your original space was linearly dependent. Features 1 and 2 are linearly dependent - they are identically correlated. In terms of the correlation matrix it will be forced to look like this (where * are not 1):

```
1 1 * *
1 1 * *
* * 1 *
* * * 1
```

This of course has implication for what you are asking, instruments with higher correlation are more likely to fit your requirements.

However, consider the following covariance matrix (of rank 2):

```
   1 1 0 0 
   1 1 0 0 
   0 0 2 2 
   0 0 2 2
```

This represents 4 instruments, 2 of which are linearly dependent but completely uncorrelated to the other 2, which themselves are linearly dependent. I would hazard that the PCA is:

```
    PC1  PC2  PC3  PC4 
f1  0    1    0    0
f2  0    1    0    0
f3  1    0    0    0
f4  1    0    0    0
```

So you can probably formulate some kind of minimisation problem associated with your stated question and return some features but in the context of the above I suspect all it is doing is finding the strongest linear dependencies.

My instinct tells me you are asking the wrong question though, since I think the answer will not illuminate what you hope to see. What is it you are trying to visualise in you data and perhaps there is a better way...

## Answer by lehalle (score 2)

https://quant.stackexchange.com/a/45691

To answer to your question on the best 1D representation of a set of vectors, I suggest to go back to the sens you want to give to "best representation":

- you may know that the PCA (principal component analysis) seeks linear combinations of your initial vectors to explain the largest fraction of their variance

- but you could use ICA (independent component analysis) if you want to obtain the most non Gaussian vector

- in fact with modern optimization tools that have been developed for machine learning, you could even design the characteristics that you want from any (non) linear combination of your vectors and obtain a "best representative component" in your custom sense.

The main question is in fact: why using one criterion instead of the other is important to you? in the case of PCA, it may be because you perceive the variance as a risk metric. It is not that much the case because it is a symmetric L2 measure.

All this to conclude that you may accept to not stock that much to a regular PCA, and add to your components a little bit of sparsity requirement. It is the topic of this PhD thesis: Interactions between rank and sparsity in penalized estimation by P.-A. Savalle. You will find all the needed maths in Pierre-André's thesis.

For you specific case, you could complete the standard PCA criterion

$$\max_{P_1,\ldots,P_{24}} \mathbb{E} \sum_n \| P_n^T F \|^2$$

by adding constraints on the sparsity of the principal components, like

$$\forall n> 1,\; \sum_d 1_{|P_{nd}|>0} \leq \rho_d.$$

This means that you ask to the optimizer to have no more than $\rho_d$ non-zero components on the $n$th component of the PCA (for all components others than the first one of course).

## Answer by Attack68 (score 2)

https://quant.stackexchange.com/a/45713

To be honest I don't believe that what you are doing is particularly useful, and I think it may even be misleading for risk management purposes. But with that disclaimer out of the way, what about this...

With your updated question we are now in a position to formulate your requirements, mathematically.

Let $E=[e_1:e_2:...:e_n]$ be your eigenvectors matrix where $e_1$ is the first PC, $e_2$ the second etc. You are searching for a binary vector, $b$ containing 0's and 1's, representing the exclusion, or inclusion, of a feature subject to the following constraints:

1) The net contribution of other PCs is zero: $b^Te_i=0 \quad \forall \quad i\ne1$ 2) The net contribution to PC1 is proportional to the number of contributing features: $b^Te_1 = \alpha b^T \delta$

The second constraint ensures you do not select instruments that are representative of PC1 in some way.

Since you cannot find this precisely you can introduce a loss function that governs whether one solution is better than another. You can even weight ($w_i$) more important considerations so that overall you have the objective function:

$$ \min_b f(b) = \sum_{i\ne1} | w_i b^T e_i | + w_1 b^T |e_i - \alpha \delta|$$

For 24 instruments where one is included ($b_i=1$) or excluded ($b_i=0$) gives rise to $2^24=16mm$, which is a little too many for a native python loop (you would have to optimise), but it can handle 15 instruments in a one second:

```
import numpy as np
import itertools

n = 15
generator = itertools.product([0, 1], repeat=n)

E = np.random.randn(n,3)
w = np.array([1,1,1])
d = np.ones(n)
alpha = 1 / np.sqrt(n)

B = np.zeros(shape=(n, 2**n))
for i, combination in enumerate(generator):
    B[:, i] = np.array(combination)

def objective_function(b, w, E, d, alpha):
    return np.sum(np.abs(np.einsum('i,ij->j', b, E[:, 1:])) * w[1:]) \
           + w[0] * np.dot(b, np.abs(E[:,0] - alpha * d))

def loop(B,w,E,d,alpha):
    f_min = 999999999
    b_min = np.zeros(n)
    for i in range(2**n):
        b = B[:, i]
        f = objective_function(b, w, E, d, alpha)
        if f < f_min and np.sum(b) > 0.001:
            f_min = f
            b_min = combination
    return (f_min, b_min)

print(loop(B, w, E, d, alpha))
```