A Complete Introduction To Time Series Analysis (with R):: ARMA processes (Part II)

Hair Parra
Nov 27, 2020 · 9 min read
Image for post
Image for post
The coefficients of the causal representation of an ARMA(p,q) process are given by the recurrence relation above.

In the last article, we saw that a general ARMA(p,q) process can be written, with the help of the autoregressive and moving-average operators as

Image for post
Image for post

. We also discussed three important properties of an ARMA process:

  1. Stationarity
  2. Causality (current observation-only depends on the past)
  3. Invertibility (can correctly represent current noise as a function of the observations from the past)

We illustrated this with the ARMA(1,1) process, and we saw that we had to put some restrictions in order to ensure that the properties above are satisfied. This time, we will generalize these ideas to the ARMA(p,q). For this, we will make use of the basics of complex numbers, so if you need a review on the essentials, you can take a look at this article.

Let’s first recall the unit disk defined on complex numbers, given by the set

Image for post
Image for post

,that is, the set of all complex numbers with modulo less or equal to 1. Graphically, this is

Image for post
Image for post

The unit circle is the same set, but holding with strict equality

Image for post
Image for post

Now, recall the ARMA(1,1) process, given by

Image for post
Image for post

From the previous article, it follows that in order to ensure stationarity, we need

Image for post
Image for post
Image for post
Image for post

How can we generalize this to the ARMA(p,q)? In simple terms, we would like to have no roots of the autoregressive Phi polynomial on the boundary of the unit circle, that is, no roots on

Image for post
Image for post

Before morving forward, we will need the following proposition:

Image for post
Image for post
Image for post
Image for post

In simple words, this proposition is saying that if the Phi polynomial is not zero, then for all complex numbers in the unit circle, we can find coefficients with an arbitrarily close distance such that the power series of the inverse Phi polynomial is convergent. The proof of this relies on analysis, so we won’t go into that here. Armed with this, we are now ready to give formal definitions/characterizations of stationarity and causality in ARMA(p,q) processes.

We generally visualize the inverse roots as they are easier to visualize inside the unit circle than outside. We will use the following graph as reference.

Image for post
Image for post
Plot of the unit disk and examples for unit

A stationary solution to an ARMA(p,q) process {X_{t}} exists and is also unique iff

Image for post
Image for post

that is if there are no roots of the autoregressive Phi polynomial on the unit circle. In the unit disk graph, and assuming we are plotting the inverse roots, only point B would violate the condition.

Image for post
Image for post
Image for post
Image for post

Equivalently, if there are no roots of the autoregressive polynomial Phi on or inside the unit circle, that is

Image for post
Image for post

If we plot the inverse roots on the complex plane like in the graph we presented before, points A and B would violate this condition. That is, we would like to have the inverse roots strictly inside the unit disk!

Finding the coefficients of a causal ARMA process

Now that we know how to determine whether an ARMA(p,q) process is causal, how can we find the corresponding coefficients of the causal representation?

Image for post
Image for post
Image for post
Image for post

where the psi coefficients are determined by the recursive equation

Image for post
Image for post

where

Image for post
Image for post

Proof

Recall that if {X_{t}} is causal, we have that

Image for post
Image for post

where

Image for post
Image for post

Note that this implies that

Image for post
Image for post

We can make use of this fact to recursively solve for coefficients of the Psi polynomial. Written more explicitly, we have

Image for post
Image for post

Note that

Image for post
Image for post

Then, we can simply start comparing coefficients on both sides of the equation yields

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

or equivalently, if the are no roots of the moving average Theta polynomial on or inside the unit circle, that is

Image for post
Image for post

Visually, we would like to have the inverse roots strictly inside the unit disk:

Image for post
Image for post
Plot of the unit disk and examples for unit

, so that in the graph above, only the inverse root A would violate invertibility conditions.

Finding the coefficients of an invertible ARMA process

Similar to before, we can also find corresponding coefficients whenever the process is invertible.

Image for post
Image for post
Image for post
Image for post

where the pi polynomial coefficients are determined by the recursive equation

Image for post
Image for post

satisfying

Image for post
Image for post

Proof

We can show this in a similar way as we did before. First, recall that if {X_{t}} is invertible, we have that

Image for post
Image for post

This implies that

Image for post
Image for post

Once again, we can make use of this to obtain the pi coefficients, by noticing that

Image for post
Image for post

and

Image for post
Image for post

Then, comparing coefficients we obtain

Image for post
Image for post

Let’s now see a concrete example using specific values for an ARMA(1,1) process. In particular, consider {X_{t}} ~ ARMA(1,1), given by

Image for post
Image for post

This implies that

Image for post
Image for post

We see that the corresponding AR(1) polynomial is

Image for post
Image for post

From which we see it has a zero at 1.25. Clearly, this falls outside the unit circle, which indicates that there exists an ARMA(1,1) process that is stationary and causal. Now, to find the causal representation of the process, we use the recursive formula

Image for post
Image for post

. Writing out the first couple of terms:

Image for post
Image for post

Which we can then, in turn, plug into

Image for post
Image for post

Similarly, the corresponding MA(1) polynomial is

Image for post
Image for post

which has a root at z=-5. This also falls outside the unit circle, which ensures that our process is invertible. Now, in order to find the inverse representation, we use the formula

Image for post
Image for post

, yielding

Image for post
Image for post

Which we can then plug into

Image for post
Image for post

One last brief example

Let’s consider the process X_{t} ~ ARMA(2,1) given by

Image for post
Image for post

Then, we have that

Image for post
Image for post

We can use the quadratic formula

Image for post
Image for post

to find the roots, yielding

Image for post
Image for post

We can then calculate the modulus for each:

Image for post
Image for post

Therefore we have no roots inside the unit circle, ensuring that our process is stationary and causal. However, notice that for the AR(1) polynomial

Image for post
Image for post

has a zero at -0.8, which indicates that the process is not invertible!

Now, we are going to simulate a couple of examples of plotting and examining polynomial roots for ARMA(p,q) processes in R. First, we import the required libraries:

Next, we are going to simulate an ARMA(2,1) process. . First, we simply create two vectors which will represent our models’ coefficients,
various ARMA models:

Next, we can numerically obtain the roots of both the AR(2) and MA(1) corresponding polynomials using the polyroot function as follows:

Note that we prepend an extra 1 at the beginning for a valid polynomial. We can now visually inspect these roots using the forecast::plot.armaroots package as below:

Image for post
Image for post
Image for post
Image for post

From these, using the conditions given before, we can see that since the inverse roots lie within the unit circle for both polynomials, we have a process that is stationary, causal, and invertible. If, for instance, one of our AR roots didn’t appear in the plot above inside the circle, then our process would no longer be causal or stationary.

That’s it for now! In the next article, we will take a look back at both the ACF and PACF, applied to ARMA(p,q) processes. Stay tuned, and see you next time!

ARMA processes (Part I)

  1. https://blog.jairparraml.com/
  2. https://www.linkedin.com/in/hair-parra-526ba19b/
  3. https://github.com/JairParra
  4. https://medium.com/@hair.parra

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…

By Analytics Vidhya

Latest news from Analytics Vidhya on our Hackathons and some of our best articles! Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Hair Parra

Written by

Data Scientist & Data Engineer at Cisco, Canada. McGill University CS, Stats & Linguistics graduate. Polyglot.

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Hair Parra

Written by

Data Scientist & Data Engineer at Cisco, Canada. McGill University CS, Stats & Linguistics graduate. Polyglot.

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store