Automating Time Series Anomaly Prediction: An intuitive guide to AutoAI on IBM Cloud Pak for Data
Time series data contains crucial information about daily business operations. Monitoring and analyzing this data can reveal valuable patterns and trends, and help predict possible anomalies or outliers. This information can then be used to identify and mitigate potential issues or problems before they occur, preventing costly downtime and other negative consequences. Time series analysis is a cornerstone for improving overall system performance by providing predictions and alerts on future anomalies. It can also help to optimize business operations for cost savings and increased efficiency.
AutoAI introduces an easy-to-use UI feature that applies several popular algorithms to analyze user’s historical time series data and understand its patterns. The resulting trained model can then predict whether there are anomalies present in new data. The identified results can then be used to issue alerts or trigger other appropriate actions.
AutoAI Time Series Anomaly Prediction is designed to leverage the normal patterns identified in the training data, and then make predictions on new data to identify anomalies that deviate from the expected or normal behavior of the training data. This technique is especially valuable for identifying anomalies in new time series datasets where only normal behavior training data is available.
In this post, we will introduce AutoAI’s new “Time Series Anomaly Prediction” (TSAP) feature, now available on IBM Cloud Pak for Data. We will take a closer look at how to use AutoAI to perform time series anomaly prediction..
Train an AutoAI experiment with demo data
As an example, we will use the electricity_usage_demo dataset to demonstrate the Time series anomaly prediction function. This dataset was generated by an open-source tool that simulates the daily electricity consumption of an industry, with some modifications made to meet our demo purposes. The data file contains two columns: a “date” column and an “industry_a_usage” column for logging daily electricity usage values. The training data does not contain any anomalies. This dataset will be used to train the AutoAI TASP pipelines.
The steps for creating a TSAP experiment are the same as for creating any AutoAI experiment: you load the training data, configure the experiment, and run the experiment to train the model candidate pipelines. For step-by-step instructions on creating a TSAP using the sample data, see this article. After adding the data source into a new AutoAI experiment, you will be led to the configuration page. If you choose “Yes” to create a time series analysis, you will see two options under the time series analysis type.
The left “Forecast” feature card is used to build a time series model on training data, which can be used to predict future values of the time series. The right “Anomaly Prediction” feature card is a new function that builds pipelines to understand the normal pattern in the training data, and then applies the pipeline to new data to determine whether a value in the new data is an anomaly or not. For an anomaly prediction case, the user only needs to provide normal time series data for training the model, which falls under the unsupervised time series anomaly prediction scenario.
The main configuration page is designed to be very simple, following the consistent style of AutoAI. Users only need to specify the feature column of interest for future anomaly prediction. If a date column is included in the data, it can be specified here as well, although it is optional. With these minimal settings, the experiment can be run directly.
Before running the experiment, let’s take a look at some details in the “Experiment settings” section.
The “Optimized metric” option on the “Prediction” tab provides the option to choose the metric to optimize for the experiment. Since the TSAP is designed to predict whether a value is an anomaly or not, it can be considered as a binary classification problem, where the model is trying to answer true/false questions. Therefore, the evaluation metrics used for binary classification can also be used to optimize the TSAP model.
In the “Algorithms to include” section, there are three types of algorithms supported in this release, totaling six pipelines. All of the pipelines generated by the experiment will be ranked based on evaluation metrics in the competition. The top three pipelines are then promoted to the Pipeline leaderboard where you can review the results of the experiment and select a winning model candidate.
With the “Holdout size” setting, AutoAI will reserve a portion of the input data series at the end for evaluating the selected pipelines. By default, 20% of the data will be used as holdout data for pipeline evaluation. Note that for this type of experiment, you cannot provide a separate holdout file; the training data must be split.
With the configuration choices complete, you can close the settings and then click on the ‘Run experiment’ button to start the experiment.
Track analysis progress with graphical UI
After initiating the experiment, you can track the progress of the analysis and training in a graphical dashboard, and then view the results.
At the experiment level, you can review the experiment summary with two views to monitor its progress. In the ‘Progress map’ view, a tree visualization shows the entire data analysis process, from the data ingestion to final pipeline generation.
In the ‘Relationship map’ view, you can explore how the recommended algorithms produce pipelines with selected feature transformations, from the inner circle representing the data to the outer circles providing insights on algorithm, pipeline, and feature transformation. This provides visibility into how the different components of the analysis are connected so you can understand the process that was used to generate the final pipeline.
During the experiment training process, AutoAI TSAP backend splits the input time series data into two parts, training data and holdout data, and the holdout data is used to evaluate and rank the suggested pipelines. To evaluate the pipeline performance, the holdout data is injected with simulated anomaly values, which allows AutoAI TSAP to test whether the selected pipeline can accurately identify these injected anomaly points. The performance of the selected pipeline in identifying these anomalies is used to rank the pipelines, with the best performing pipeline being recommended as the best solution.
Review the recommended pipelines along with insights
During the training process, completed pipelines are added to a pipeline leaderboard, and top pipelines are listed under the Experiment summary view. The final “Pipeline leaderboard” displays the winning pipelines. TSAP supports 4 types of anomalies: level shift, trend, localized extreme, and variance. These anomalies cover typical time series anomaly situations.
AutoAI injects these 4 types of anomalies separately into the holdout data and then checks how each pipeline performs on the holdout data with these 4 types of anomalies. In the “Pipeline leaderboard” table, you can find how these pipelines performed on the holdout data with the specified metric.
Based on these metric values, the “Aggregate score” is calculated by aggregating the optimized metric values for the 4 types of anomalies. The score for each pipeline is ranked using the Borda count method and then summed to get the aggregate score.
You can click on a pipeline row to view its details.
The ‘Model evaluation’ view displays a table that shows how the pipeline performs on the 4 anomaly types, as measured by 5 evaluation metrics. This can help you understand the pipeline’s strengths and weaknesses.
Additional information can be found in other tabs within Pipeline details page to help you better understand the build process and results of the selected pipeline.
Predict anomaly in new data
After selecting a recommended pipeline from the leaderboard, you can save it as a model in your project and deploy it for prediction. The model can be promoted to a deployment space, where an online deployment can be created, as shown in the following image.
In the online deployment page:
· The ‘API Reference’ tab provides code snippets in various programming languages that demonstrate how to access the deployed model for online prediction.
· In the ‘Test’ tab, you can quickly input new data in the form of a table or as JSON and obtain a prediction from the deployed model. You can also upload a CSV file in spreadsheet form to provide input values for anomaly predictions.
Click on the “Predict” button to view the anomaly predictions on the new input time series data. You will see that a few data records are predicted as anomalies and marked as “-1”. This demonstrates how AutoAI brings convenience to the process of building pipelines and scoring new data for anomalies with just a few clicks.
Summary
In this article, we have walked through the key steps involved in performing automatic time series anomaly prediction using AutoAI in IBM Cloud Pak for Data. Using a demo dataset, we have demonstrated how easy it is to train time series anomaly prediction models with candidate pipelines using AutoAI. The analysis process and progress displayed in a graphical user interface. We have also explained the logic used to evaluate and rank pipelines, the pipeline leaderboard and detail pages that display experiment results, and how to predict anomalies in new data. We hope that AutoAI proves to be a valuable tool for your data science exploration and project development.
There is also a blog available for playing with TSAP in a notebook using AutoAI Python API, which you may find useful as a reference.
I believe you are now ready to experience our new feature in AutoAI. Please go to IBM Cloud to embark on your journey.
Acknowledgement
I would like to thank Julianne Forgo, Sier Han, and Jun Wang for their review and helpful revisions.
