Stop using Moving Average to smooth your Time Series
When it comes to time series analysis, the art of smoothing data is critical for revealing underlying trends that are often obscured by noise. Two popular techniques stand out in the data scientist’s toolkit: Moving Averages and Savitzky-Golay Filters. While Moving Averages have long been the go-to method for a quick-and-dirty smoothing, it’s the Savitzky-Golay Filter that often takes the crown for a more nuanced and accurate portrayal of data trends.
In this article I will provide an explanation, accompanied with an example with Python code, on why Savitzky-Golay Filters often make a better job smoothing Time-Series.
The Essence of Time Series Smoothing
Imagine you’re sifting through sensor data trying to detect a meaningful pattern, or you’re analyzing stock prices to forecast future movements. The raw data is a rollercoaster of ups and downs, rife with random variations that make your task akin to finding a signal in static. Smoothing helps by dampening these fluctuations and teasing out a clearer signal.
Loading the Time-Series
The first step in our analysis is akin to load the time series. But of course we begin importing necessary libraries.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats
from scipy.signal import savgol_filter
import plotly.express as px
from statsforecast import StatsForecasttrain = pd.read_csv('https://auto-arima-results.s3.amazonaws.com/M4-Hourly.csv')
test = pd.read_csv('https://auto-arima-results.s3.amazonaws.com/M4-Hourly-test.csv').rename(columns={'y': 'y_test'})
uid = np.array(['H386'])
df_train = train.query('unique_id in @uid')
df_test = test.query('unique_id in @uid')
StatsForecast.plot(df_train, df_test, plot_random = False, engine='plotly')
The time series is taken from from the M4 competition dataset and I chose this one because it has a recurrent (seasonal) yet not smooth behavior.
I used StatoForecast.plot function because of its ease of use (and also because I copied and pasted this line of code from the previous article ;P)
Smoothing the Time-Series
In the realm of time series smoothing, the concept of ‘window size’ plays a pivotal role in shaping the outcome of our analysis. Both the moving average and the Savitzky-Golay filter are governed by this parameter, which determines the extent of data considered for smoothing at any given point. Think of window size as the aperture of a camera lens — the wider it is, the more it captures, affecting the resultant image’s clarity and detail. In a moving average, the window size defines the number of data points averaged to produce a single smoothed point. For the Savitzky-Golay filter, this size not only averages but also fits a polynomial to the data within the window, striking a balance between smoothing and maintaining the fidelity of the signal’s features¹. If you are interested in how the Savitzky-Golay filter in details, you can read the sources in the reference section. Below the code:
computed_features = [] # I will need this list to plot later the smoothed series
for window_size in [10, 25]:
df_train.loc[:,f'moving_average_{window_size}'] = df_train['y'].rolling(window=window_size, , center=True).mean()
df_train.loc[:,f'savgol_filter_{window_size}'] = savgol_filter(df_train['y'], window_size, 2)
computed_features.append(f'moving_average_{window_size}')
computed_features.append(f'savgol_filter_{window_size}')We computed the total of 4 smoothed versions of the original time series, using to window sizes 10 and 25. Let’s now see the results for the first window size compared to the original Time Series.
fig = px.line(df_train[df_train.ds>500], x='ds', y=['y'] + computed_features[:2], title='Different moving average estimators',
labels={'Value': 'y', 'Date': 'Date'},
line_shape='linear')
# Improve layout
fig.update_layout(
xaxis_title='Date',
yaxis_title='Sensor Value',
hovermode='x'
)
fig.show()
The moving average, flows smoothly but it fails to incorporate the first smaller peak if followed by an higher peak. The Savitzky-Golay filter, on the other hand, maintains a closer step with the time series, preserving peaks and troughs with precision. As shown after, when increasing the window size, the Savitzky-Golay filter incorporate this information by anticipating the peak. We can appreciate that despite being its most appealing feature, moving average’s simplicity is also its Achilles’ heel.
The Moving Average can be slow to respond to real shifts in data, often lagging behind when the trend changes direction. Moreover, it treats all points in the averaging window equally, ignoring the nuanced differences in their relevance. In contrast, the Savitzky-Golay filter By fitting successive subsets of adjacent data points with a low-degree polynomial by the method of linear least squares, the Savitzky-Golay filter maintains the shape and features of the underlying data¹. This means it preserves important characteristics like peaks and troughs, which are often smoothed out by the moving average.
The Second Movement: Window Size of 25
As we saw in the previous plot, the Savitzky-Golay filter still reported the peaks and may be the case where we would like to eliminate them. So let’s see if increasing the window size we can get this result.
fig = px.line(df_train[df_train.ds>500], x='ds', y=['y'] + computed_features[2:4], title='Different moving average estimators',
labels={'Value': 'y', 'Date': 'Date'},
line_shape='linear')
# Improve layout
fig.update_layout(
xaxis_title='Date',
yaxis_title='Sensor Value',
hovermode='x'
)
fig.show()
Here, the Savitzky-Golay filter did an excellent job capturing the seasonality of the time series with no delay and removing the spikes, while the moving average focused all its attention to the long term average, losing many pieces of information contained in the signal.
Conclusion
Overall, the Savitzky-Golay filter tends to maintain higher fidelity of the signal² while removing the unnecessary spikes when the window size is properly tuned. Anyway, the moving average still serves its purpose of computing the average of the time series througth time, even though the same result can be obtained enlarging the window size of the Savitzky-Golay filter (and maybe with better precision), one could evaluate to use it if the interest is in capturing the underling mean around which the process revolves. But for most of the smoothing use cases, the Savitzky-Golay filter does a much better job.
I hope you enjoyed reading this article as much as I enjoyed writing it. Please reach out with any comment because any feedback helps.
I wish you good learning
Roberto
References
[2]: Kawala-Sterniuk, A.; Podpora, M.; Pelc, M.; Blaszczyszyn, M.; Gorzelanczyk, E.J.; Martinek, R.; Ozana, S. Comparison of Smoothing Filters in Analysis of EEG Data for the Medical Diagnostics Purposes. Sensors 2020, 20, 807. https://doi.org/10.3390/s20030807
