Time Series Forecasting using Kalman Filter
A theoretical explanation for the Kalman filter algorithm for real-world estimations
Time Series forecasting is a difficult concept to grab in. We have a number of different model families that can be used to forecast time series be it statistical models like AutoRegression, ARIMA, Holt-Winters, etc, or Neural Network-based solutions like LSTMs. There exist one more such family of solutions i.e. State Space models.
My debut book “LangChain in your Pocket” is out now !!
If you are a vlog person
What are State Space models?
State Space models, in layman's terms, are more associated with the real world & consider the errors in calculation/ecosystem dynamics that can happen while measuring which isn’t considered by any other modeling family. Such models work on the ideology that
- You can never measure the true value of any system. For example, If you measure your height using some scale, the scale is prone to have some rate of error & its measurement aren’t 100% exact
- We can infer the true value using measurements. So, if you measure your height using some scale, we can use the measurement, to reach the true value by using prior knowledge and considering error rates
Kalman Filter is amongst the most popular & widely used algorithms not just for time series forecasting but estimating true states in a lot of real-world applications like NASA’s Apollo 11 mission (The one Neil Armstrong onboarded) to estimate the flight’s actual position considering all the dynamics & errors in the ecosystem
I hope this is enough motivation for anyone to know about this super important algorithm
Before we move ahead, where can we use the Kalman filter?
When our system looks something like the below equations
Xₜ = F x Xₜ₋₁ + B x Uₜ + Wₜ
Yₜ = A x Xₜ + Vₜ
Where,
Xₜ = Actual value
Yₜ = Measured value (requires correction)
B x Uₜ = is some external force/factor in the system (usuallu taken as 0, B is constant)
Wₜ = Stochastic error i.e. random error term
Vₜ = measurement error
So, if our problem statement satisfies the 2 equations, we are good to go ahead & apply the Kalman filter. For example: Calculate the exact position of a satellite where Y becomes the position estimated by sensors & X becomes the exact position.
Kalman filter involves 3 steps done back & forth i.e. prediction, filtering over the predicted values (sort of correction) followed by updating parameters used. The algorithm has 5 equations
Predictions
Xₚₜ = F x Xₜ₋₁ + B x Uₜ
Pₚₜ = F x Pₜ₋₁ x Fᵀ + Q
Calculate K
Kₜ = Pₚₜ x Aᵀ x (A x Pₚₜ x Aᵀ + R)⁻¹
Filtering
Xₜ = Xₚₜ + Kₜ x (yₜ — A ×Xₚₜ)
Pₜ = (1-Kₜ x A) + Pₚₜ
Lots of things to clarify !!
- Xₜ is the term to be forecasted at timestamp t
- P is the error covariance matrix (not much confident about this term)
- ₚₜ subscript is used to denote predicted terms (they aren’t the final terms, predicted + filtering gives us the final terms)
- Kₜ is Kalman Gain at timestamp t
- A, F, and B are all constants
- R is measurement error variance (variance of Vₜ)
- Q is stochastic error variance (variance of Wₜ)
- Yₜ is the measured value
Before we begin with the process of estimating Xₜ, we need to know the below things
- Value X₀, P₀, and Y or Vₜ. Even assuming X₀, P₀ initially can help
- The idea about Uₜ (though, usually taken 0)
- The constant variables like F, A, and B alongside R & Q
Before we wrap up, a big disadvantage we can see with the algorithm is too many hyperparameters to tune in, which in general, is a problem with most state space models. But this is obvious as this model family is closest to real-world dynamics. Most other model families consider very ideal conditions & assumptions before making a sort of forecast, making them error-prone in real-world scenarios & hence inaccurate forecasting.
How can we use the Kalman filter for time series forecasting where we don’t have any measurement Y? I think (couldn’t find much on this ground) we can use the Kalman algorithm in combination with some other forecasting algorithm where predictions from that algorithm (say ARIMA) can be considered as Y & the filtered X can be calculated using it.
Interestingly, python has a library filterpy that gets you covered for Kalman filter implementation. You can find the sample run here
A great resource: http://bilgin.esme.org/BitsAndBytes/KalmanFilterforDummies
