Cloud with a chance of meatballs

Wouldn’t you love to see into the future? Know exactly what will happen next week or next year? Well, until Marty McFly takes you to the year 3000 (and back) in his DeLorean, or you suddenly shape shift into That’s So Raven, you will have to settle for seeing the future through the eyes of a forecast.

More accurately you will have to settle for looking at the future of forecasting through my eyes! I’m a software engineer working in the Innovation Team in Kainos. Our team works to investigate emerging tools and technologies, discovering the risks and rewards along the way and ultimately sharing our knowledge.

Recently we have been investigating the growing number of forecasting tools. specifically we have been investigating the growing number of cloud hosted forecasting tools!

What is a forecast?

Forecasting is a prediction, projection or estimate of a future event or occurrence. For many years businesses have used forecasting to predict future outcomes and make informed business decisions. Making predictions on future demand and expected sales can help companies plan things like;

• How many staff to hire
• How much inventory to purchase.

The techniques used to create these forecasts range from simplistic ideas to more complex statistical algorithms.

Forecasting Methods

Forecasting methods can be broken up into 2 major categories: Qualitative and Quantitative. Qualitative Forecasting focuses on human judgement and less on data. We base these forecasts on opinions, intuition and personal experience. They are subjective in nature and don’t rely on mathematical computations. Examples include market surveys in which we interview customers to judge their preferences or the Delphi method where a consensus is agreed upon among a group of industry experts. Qualitative Forecasting methods are best used when there is a lack of appropriate historical data.

Quantitative Forecasting relies heavily on mathematical models. We can further break quantitative methods up into two groups:

1. Time-Series Models that look at past patterns of data and attempt to predict the future based on those patterns.
2. Associative Models that look at how other things happening in the environment might cause a change in future measures.

A time-series model could accurately predict a seasonal rise in retail sales over the Christmas period, whereas an associative model might do better at predicting a rise in ice-cream sales during a heatwave.

Unfortunately for us, there is no such thing as a one size fits all forecasting model. The choice of model depends on several factors including data type, data amount and how far into the future we want to forecast.

The Cloud

2020 was the year the world woke up to the importance of the cloud, and we’re not surprised. When our computing and storage takes place on servers in a data centre, instead of on the user device we see tremendous benefits including flexibility, security and cost savings to name but a few.

Amazon Web Services (AWS) is the leading cloud vendor. They control 32% of the market with customers in 190 countries. As such, we have focused in on AWS and this post will explore 3 of their most popular forecasting services. Evaluating how they perform and some of their strengths and weaknesses.

Daddy, what are clouds made of?

Linux servers, mostly.

The Experiment

To put these cloud-hosted forecasting services to the test, we use each of them to predict the sales of Walmart stores over a 6-month period.

We used a Kaggle dataset that provides sales figures for 45 Walmart stores and the temperature, fuel price, unemployment rate and Consumer Price Index (CPI) in the area on the date the sales were recorded. We tested 3 AWS offerings: AWS Forecast, and Sagemaker with the DeepAR and XGBoost algorithms.

Of course, we want to find the most accurate forecast, but there are a few other things we want to consider when pitting these services against each other:

• How suitable is it for working with time series?
• Is this service easy to use?
• How quickly can it train or deploy a model?
• How accurate are the model predictions?

Let’s see how they hold up …

AWS Forecast

Amazon Forecast is described as a fully managed service that uses machine learning to deliver highly accurate forecasts. Amazon Forecast allows users to choose between 5 algorithms, including Facebook’s highly regarded Prophet algorithm that wouldn’t be available on AWS Sagemaker. It can deal with data that is very granular and can take in up to 100 million records.

However, one of the biggest selling points with this service is the sleek interface which makes it accessible to users with zero programming experience. Simply import your dataset, choose an algorithm to use and train a predictor. Or better again, don’t choose an algorithm. Forecast’s ‘AutoML’ feature will run through all possibilities until it finds the best results for your problem. Easy, right? Well, like most things, Forecast has its pitfalls.

“sleek interface which makes it accessible to users with zero programming experience”

Forecast is extremely rigid in how it will ingest your data. Unless it’s in the perfect format and ordering the service will reject the data with little or no explanation why. From personal experience, we advise you do not use the ‘AutoML’ feature. At first glance, it seems like a golden hammer to solve all your forecasting problems. In reality, the hammer will be old and rusted by the time we train this predictor as it takes SO LONG! But if you are willing to wait for Forecast to run through every possibility to find the best one, the predictions it can be pretty accurate.

DeepAR

DeepAR is a machine learning algorithm for time series forecasting that makes use of recurrent neural networks. Neural networks work by mimicking the interconnectivity of the human brain. DeepAR outperforms statistical models like ARIMA and Exponential Smoothing by training one model that understands the similarities between many time series, where the statistical model would require a new model for each time series.

Because DeepAR was designed specifically for time series forecasting, we can cut our feature engineering time in half. Unlike many algorithms that are used for time series, DeepAR can also consider dynamic and categorical features in the dataset. DeepAR can even make predictions on time series that weren’t in the training dataset. Not to mention the fact that the algorithm automatically considers lag data when training the model and making predictions.

“DeepAR is fussy! The training data must be in the perfect format!”

With all of these excellent features, you might think that DeepAR must be the perfect forecasting solution, but it has its drawbacks: DeepAR is fussy! The training data must be in the perfect format, not exactly easy to achieve. The prediction length is rigid, once set in the training we cannot change it. When testing the Walmart data, we set the model to make 42 weeks predictions, the longest prediction it could make with the dataset. Now we can’t simply check the forecasted sales for next week, we have to get the next 42 weeks.

The DeepAR algorithm performed similarly to XGBoost, with some stores achieving better results than others. However, DeepAR is predicting farther into the future, so we’re inclined to think it’s a bit more powerful.

Sagemaker XGBoost

Amazon Sagemaker is a fully managed machine learning service. With Sagemaker, developers can build and train machine learning models and deploy them to a production-ready hosted environment. Sagemaker allows you to bring your own model as a docker image or developers can make use of one of Amazon’s built-in algorithms such as XGBoost.

XGBoost is a popular open-source machine learning algorithm that is decision-tree based. It was developed as a research project at the University of Washington and it uses a gradient boosted framework. XGBoost is a robust machine learning algorithm, but it wasn’t built for forecasting and it doesn’t understand time-series data.

“ it wasn’t built for forecasting and it doesn’t understand time-series data”

So if this model doesn’t understand how to deal with time series, why are we trying to use it to make predictions on time-based data? Well, XGBoost is one of the top-performing models available. It can handle large datasets with many features, and it has great execution speed and accurate performance without being prone to overfitting. We can transform our time series data into a supervised learning model. This can be a time-consuming process, but will it be worth it for an accurate forecast? Let’s find out …

To transform this from a time series problem to a supervised learning problem, we replaced the date column with two alternatives. The ‘Days_Passed’ column counts the number of days from the beginning of the dataset. This feature allows the model to understand the major trends in the data. The ‘Day_Of_Year’ column, as the name suggests, gives the day of the year as a number from 1 to 365. This teaches the model annual seasonality in the data. For example, most years Christmas falls on the 359th day of the year and we would notice a peak in sales every year around that time.

After some trial and error, we tuned the hyperparameters and trained the model. We trained the ultimate model on data from a small subset of the 45 stores that all followed similar patterns. Despite clustering the data, the results were still mixed. As seen in the graphs above, the model created some very inaccurate predictions and others that were almost perfect. A 6-month forecast for store 3 had a mean error of over 18%, whereas Store 31 had less than 3% error.

Take-away

Ultimately, what is the best tool? Which one will create a forecast to give the MET office a run for their money? For me the winner is XGBoost in Sagemaker, although Forecast comes a close second. Despite being difficult and time-consuming to pre-process the dataset, XGBoost is the fastest to train, makes flexible forecasts, and creates the most accurate predictions.

If you enjoyed this blog make sure to check out the Innovation team’s publication!