Interpreting PCA Regression Betas Under Multicollinearity

# Orthogonal Regression/PCA


# Orthogonal Regression/PCA












I am doing orthogonal regression. My X matrix consists of returns on a broad market index, value index, growth index, a few sectors,.....(my Y is the returns on an equity fund)

I am regressing the Y on the (1st two) principal components of X (this is to avoid the problem of multicollinearity in X). I then back out the betas for the original X variables by the matrix multiplication of the eigenvector matrix and the betas for the principal components. All good so far

I wanted to make sure everything was right so decided as a test to regress the returns of the broad market index on X (and remember that the broad market index returns are actually in X). I would expect the beta for the broad market index to be very near 1 but when I do that orthogonal regression it is not - it is similar in value to all the other betas (around 0.15).

This surely does not make sense right? When I do plain old regression of the returns of the market index on X (which would suffer from multicollinearity given the high correlation amongst the X variables, right?), the beta estimate is exactly 1 (and the betas for the other factors are very small in comparison), but when I use orthogonal regression the beta is 0.15.

Is the small beta for the market index factor not a concern?

## Answer by Acoustesh (score 1, accepted)

https://quant.stackexchange.com/a/19424

If X contains several highly correlated indexes, the first PCA will be a linear combination of them and its weights will be similar because at the end they represent the same underlying phenomena. When you do a regression with the same variable in Y and X you will have perfect match of that specific regressor by construction.

The real problem of colinearity is that many different linear combinations of your variables X gives you very similar results. PCA only restricts the possibilities of those combinations to get more stability in your model parameters, but it does not mean that those are "the" real parameters and there are many other combinations that will give you similar results, therefore betas are not so easy to interpret.

My suggestion is that you build a model with less colinearity by construction. For example you could use a broad stock index, like a world index weighted by country GDP or market cap (let's call it I1), then find the projection of X in into the subspace orthogonal to I1 (let's call it X') to eliminate the broad market effect in each index of X. Doing that you will eliminate most colinearity problems, unless you use similar indexes in X. In that way your model would have two kinds of betas. One associated to broad market movements (the one associated to I1) and the rest associated with specific sectors, styles,countries, etc. One way to easily find X' is to construct It using the Regresion residuals of each index in X using I2 and constant as the only regressors. X' will be the matrix with the residuals of each individual regressions. Then you use I2 and X' as regressors. If you avoid X' indexes too similar (like using oil and energy) you will avoid multi colinearity issues and will be able to interpret your results easily. If your X has many similar variables I would consider using PCA on the X' values, but again it could be difficult to interpret.