Why Full Rank Matters in Generalized Linear Model Design Matrices
Summary
The document asks why a generalized linear model’s design matrix is assumed to have full column rank and whether maximum-likelihood estimation can proceed when it does not. The response connects rank deficiency to non-identifiability: if columns are linearly dependent, distinct parameter vectors can produce the same linear predictor, so the coefficients may not be uniquely determined.
It also explains the role of the Fisher information matrix in large-sample parameter uncertainty and in common iterative estimation methods such as scoring. Rank deficiency can make this matrix singular, preventing its ordinary inverse from defining the usual covariance estimate or update step. The answer gives a concise linear-algebra rationale, but it does not discuss alternative remedies such as dropping redundant predictors, constraints, or regularization, nor does it distinguish cases where fitted values remain estimable even though individual coefficients are not.
Key ideas
- A full-column-rank design matrix avoids linear dependencies among predictors and supports unique coefficient identification.
- Rank deficiency can make the Fisher information matrix singular.
- The usual inverse-based covariance estimate and scoring update may fail when information is singular.
- Model predictions can remain meaningful even when some individual coefficients are not uniquely estimable.
Tags
Full text
# Full rank design matrix GLM
# Full rank design matrix GLM
I have a question regarding Generalized Linear Models and its design matrix. In the GLM framework we say that $h(E(X))=Z\beta$, where $Z\in \mathbb{R}^{M\times (r+1)}$ is called a design matrix, that we assume has full rank. $h(\cdot)$ is the canonical link and $\beta$ is the parameter vector.
Now, why does the design matrix Z have to have full rank? The MLE for $\beta$ is obtained by solving the following equation: $ZVb'(Z\beta)=Z'VX$. Is this equation numerically not solvable for a design matrix that does not have full rank?
## Answer by Carson McKee (score 1)
https://quant.stackexchange.com/a/61192
A standard result in linear algebra is that a matrix $X$ is invertible if and only if it is full rank.
Now for a large sample size, the approximate distribution of the ML estimator of the GLM parameters is obtained as, $$ \widehat{\beta} \sim \mathcal{N}(\beta, \mathcal{I}^{-1}), $$ where $\mathcal{I} = X^{\top}WX$ is the Fisher Information evaluated at the MLE. $W$ is a diagonal matrix with elements $w_{ii} = \frac{1}{var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i} \right)^2$, $\mu_i = E(Y_i)$ and $\eta_i$ is the linear predictor for sample $i$.
So if your design matrix $X$ is not full rank, then it is not invertible. Therefore, $\mathcal{I}$ is not invertible, since $\mathcal{I}^{-1} = (X^{\top})^{-1}W^{-1}X^{-1}$. This would result in an undefined variance/covariance structure for $\widehat{\beta}$. In addition, one of the most popular methods of finding $\widehat{\beta}$ is by the method of scoring, $$ \beta^{(r+1)} = \beta^{(r)} + \mathcal{I}^{-1}\frac{\partial l}{\partial \beta^{(r)}}. $$ Clearly we cannot score if $\mathcal{I}$ is not invertible.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.