Why a Kalman Filter Covariance Can Approach Constant Correlation
Summary
The document investigates why correlations derived from a Kalman filter’s posterior covariance estimate for a commodity forward curve appear nearly constant across contracts. The described setup uses a latent state containing forward values and drift estimates, with process-noise covariance based on average variance and observation-noise covariance based on market-consensus uncertainty. The question is whether the uniform off-diagonal correlations reflect a property of the filter or a modeling issue.
The accepted answer explains that a time-invariant linear system can converge to a steady-state Kalman gain and updated estimation-error covariance when the state transition is stable, specifically when its spectral radius is below one. Such convergence can help explain a covariance estimate that stops changing over time, though the response does not show that it explains the reported correlation level or diagnose the user's particular matrices. The answer supplies a general condition, not a full investigation of the model or data.
Key ideas
- A time-invariant linear state-space model can converge to a steady-state Kalman gain.
- The updated estimation-error covariance can also converge under stable system conditions.
- The stated stability condition is that the state transition matrix has spectral radius below one.
- Steady-state convergence alone does not establish the cause or value of the observed correlations.
- The response does not inspect the specific process and observation covariance matrices.
Tags
Full text
# Odd Result from Computing Correlation Matrix from Kalman Filter Posteriori Covariance Estimate
# Odd Result from Computing Correlation Matrix from Kalman Filter Posteriori Covariance Estimate
I am using a Kalman Filter to estimate the return dynamics of a forwards curve on a particular commodity. My state space is the initial forwards values, and an initial guess of the drift functions for each forward (initially just the first observed price change). These are for OTC products, my estimates for uncertainty are based on market consensus standard deviations. My Estimate of the Q matrix (process covariance noise) is based on the average variance, and my estimate for the R matrix (covariance observation noise) is based on each days market consensus standard deviation.
From the resulting Posteriori Covariance matrix, I estimate the correlation matrix. The result is constant correlation. Basically the off diagonal is roughly 72% almost everywhere. I am not sure why this should be the linearly optimal result, or perhaps there is a flaw in my understanding of the Kalman filter?
I can not provide the data I used for this experiment, but I am more than happy to answer clarifying questions if it would help the community to better answer my question: Particularly, what could be the cause of observational constant correlation?
## Answer by Kermittfrog (score 1)
https://quant.stackexchange.com/a/60712
Let your linear system be defined by the latent multivariate state variable $x_t$, progressing in an AR(1) fashion, and let your observation at time step $t$ be linear in the latent state:
$$ \begin{align} x_{t+1}&=Ax_t+u_t\\ y_{t}&=Hx_t+v_t\\ \end{align} $$ with $u_t\sim N(0,Q)$ and $v_t\sim N(0,R)$ time-and-state-independent process noise.
Let $K_t$ denote the Kalman gain and $P_{t,t}$ denote the updated estimation error covariance.
As this system is time-invariant ($A,H,Q,R$ are constants), it will result in
- a steady state Kalman Gain $K_t\to K_{\infty}$
- a steady state updated estimation error covariance $P_{t,t}\to P_{\infty}$
for $t$ sufficiently large* and whenever the spectral radius of $A$ is smaller than 1.
Below, you find a couple of references (... I googled them, maybe some more thorough research is needed on your end):
A quite recent introduction to the KF
Search for 'steady' in here
Section 2.2 of these leture nodes
* Sufficiently large means usually some small number; in my experiments the filter usually converges after 10 steps or so.
HTH?Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.