Time-Series Forecasting with Cortex™ and Prophet
Using Facebook’s Prophet library to predict the flow of critical threats, as featured at the Palo Alto Networks Ignite ’19 conference in Austin, Texas.
Maybe you’ve experienced it before — that feeling when you’re deeply interested in learning about a new topic but honestly have no idea where to begin.
Such was the case when I first heard the words “Machine Learning” (ML) and “Artificial Intelligence” (AI) mentioned at Palo Alto Networks. It wasn’t just fear of the unknown. There was a real sense that these concepts could radically alter our digital way of life. It was intimidating, especially for someone coming from an IT and network security background. It felt counterintuitive to embrace something that could lead to my own workplace obsolescence (a tension so intriguing to me that it led to an Ignite ’18 talk titled “How to Automate Yourself Out of a Job (and why you should)” which had a more optimistic take on workplace obsolescence).
So, with Ignite ’19 fast approaching (and ideas for live demos running thin!) I decided it was time to plunge right into the AI/ML abyss. It began with a simple question: “Could security posture be forecasted like the weather?” Armed with the Cortex™ Data Lake and the CDL Python SDK, I set out knowing that I, at least, had the data. All that remained was to find a suitable ML methodology for conducting my experiment.
Finding the Right Tool
I was attempting to predict the flow of critical severity threats in my environment which reminded me of weather and stock market forecasting.
I quickly learned that, for a beginner, it’s easy to be led astray, especially given the multitude of libraries and tools available and the many blog articles and tutorials written for them.
It’s difficult to know where to begin without a well-defined hypothesis in mind, so my suggestion here is to think deeply about the particular question you wish to answer and then find a way to ask that question in the simplest way possible. Better yet, consider whether your question has been asked before. In my case, I was attempting to predict the flow of critical severity threats in my environment which reminded me of weather and stock market forecasting. So, that’s how I framed my search.
Enter Facebook Prophet
Behold. Time-series forecasting in a few lines of code!
After a few dozen searches and a few hours of Tensorflow and PyTorch tutorials, I (conveniently) determined that a Recurrent Neural Network (RNN) might be overkill for the particular dataset I was working with. More importantly, diving in RNNs for me, at the time, meant spending more hours of my life in the “valley of suck.”
No. I needed something quick and easy and my dataset consisted solely of two columns: datestamp (hourly) and the number of critical severity threats observed at each interval (see Fig 1).
When I first encountered Facebook’s Prophet library I almost couldn’t believe it. How could this library not be in the top results when searching for “time series forecasting with python”? A quick glance at the quickstart left me feeling like I was finally on the right track, especially since Prophet appeared to abstract and simplify the more tedious sci-kit learn tutorials I had already become familiar with.
Behold. Time-series forecasting in a few lines of code!
Too good to be true?
Forecasting the flow of critical severity threats is a lot like trying to predict the flow of the stock market
If you’re skeptical, it’s for a good reason. Forecasting the flow of critical severity threats is a lot like trying to predict the flow of the stock market, although even that might be easier than you think. Still, if you’re looking for an entry point into AI/ML, time-series forecasting seems to be a great introduction — at least it was for me.
If you’re interested in replicating the experiment I packaged and published it as a Google Colaboratory notebook which you can find below:
Until next time, happy threat wrangling!
