EM Estimation for a Conditional Gaussian Model with Missing Data
Summary
The document describes an expectation-maximization approach for estimating a conditional Gaussian model when observations of either variable may be missing. It frames the conditional relationship between two variables as a form of a bivariate Gaussian and explains that the E-step should use posterior distributions for missing values, rather than filling them with posterior means alone. Expected first moments, squared moments, and cross-products form the sufficient statistics used to update the mean vector and covariance matrix.
The answer illustrates this with a small example containing observed and missing entries, and notes that inference methods such as a junction tree or Kalman filter can supply posterior means and conditional variances. The M-step updates the joint covariance, from which conditional regression parameters and conditional variance can be derived. The explanation is conceptual and terse: it does not fully spell out the general E-step for each missingness pattern or discuss convergence, model assumptions, or numerical stability. Its example is not evidence of performance on broader datasets.
Key ideas
- Representing the conditional Gaussian as a joint bivariate Gaussian provides a route to estimating its parameters.
- The E-step must account for posterior uncertainty in missing values when computing squared moments.
- Cross-moments between the variables are needed to update the covariance matrix.
- The updated joint mean and covariance can be converted back into conditional Gaussian parameters.
- The answer sketches the procedure but leaves implementation details and convergence considerations unspecified.
Tags
Full text
# EM for conditional Gaussian model
# EM for conditional Gaussian model
Let $$X_1\sim N(\mu_{X_1},\sigma_{X_2}^2)$$ $$X_2\sim N(\mu_{X_2}, \sigma_{X_2}^2)$$ where $\mu_{X_2}=c+aX_1$. Also, I have data $D$ (with missing values on $X_1,X_2$).
How can I update/estimate the parameters $\mu_{X_1},\sigma_{X_1},\mu_{X_2},a,c,\sigma_{X_2}$ using EM? i.e. what is the formula for updating $\sigma_{X_2}$?
My model is a conditional Gaussian, which is a conditional form of a bivariate Gaussian $(X_1,X_2)$ with mean vector $(\mu_1,\mu_2)^\top$ and covariance matrix $$\left( \begin{matrix} \Sigma _{11} & \Sigma _{12} \\ \Sigma _{21} & \Sigma _{22} \\ \end{matrix} \right)$$
Here is a reference to convert bivariate Gaussian to conditional Gaussian: $$\mu_{2|1}=\mu_{2}+\Sigma_{21}\Sigma_{11}^{-1}(X_1-\mu_1)\quad,\quad \Sigma_{22|1}=\Sigma_{22}-\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}$$ which yields my model.
It seems that when $X_1$ has different observations and $X_2$ is unobserved, the variance of $X_2$ remains unchanged.So how to update the $\sigma_{x_2}$? Do I need to estimate covariance matrix?
Initial setting for the model $$X_1\sim N(5,7)\quad,\quad X_2\sim N(0.5X_1,8)$$
Data:
$$X_1:\operatorname{9\,\,4\,\,NA}$$
$$\quad X_2:\operatorname{NA\,\,NA\,\,3}$$
## Answer by snowave (score 0, accepted)
https://quant.stackexchange.com/a/28307
When $X_1$ is unobserved, at iteration $k=1$ of EM, the posterior mean value (when $X_2=3$) is $5.18$ by using an inference algorithm, i.e. Junction tree/Kalman filter. Then the sufficient statistics for $X_1$ is:
$s_1=\Sigma_{i=1}^nx_{i1}=9+4+5.18$ and $s_{11}=\Sigma_{i=1}^nx_{i1}^2=9^2+4^2+(5.18^2+\sigma_{11.2})$
where $\sigma_{11.2}$ is the posterior conditional variance of $X_1$ given $X_2$, when $X_1$ is unobserved.
$s_2=\Sigma_{i=1}^nx_{i2}$ and $s_{22}=\Sigma_{i=1}^nx_{i2}^2$. When $X_2$ is unobserved a term $\sigma_{22.1}$ is also added to $x_{i2}^2$.
So parameters $\mu_{X_1}$ and $\sigma_{X_1}$ are obtained. Similarly, posterior mean of unobserved $X_2$ is also obtained by using i.e. Junction tree inference.
The sufficient statistics for $s_{12}=\Sigma_{i=1}^nx_{i1}x_{i2}$, and $s_{21}=\Sigma_{i=1}^nx_{i2}x_{i1}$.
Mean vector is then: $\mu^{(k)}=(s_1/n, s_2/n)$.
Covariance matrix is: $\Sigma^{(k)}=$$\left( \begin{matrix} \Sigma _{11} & \Sigma _{12} \\ \Sigma _{21} & \Sigma _{22} \\ \end{matrix} \right)$$=$$\left( \begin{matrix} s_{11}/n-(s_1/n)^2 & s_{21}/n-(s_2/n)(s_1/n) \\ s_{12}/n-(s_1/n)(s_2/n) & s_{22}/n-(s_2/n)^2 \\ \end{matrix} \right)$$ $
Other parameters in the question are obtained. Conditional mean and conditional variance of $X_2|X_1$ is calculated by the formula in the reference of the question. I have validated these formulae with Matlab the result is correct.Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.