Billings and Revenue Forecasting—Part 1
It was 2017 and a very exciting time at Pluralsight. Sales were growing rapidly, new employees were coming in left and right. If you had 6 months of tenure at the company, you were considered experienced. One of the biggest things driving the excitement was the prospect of a soon to be announced IPO. Transitioning from a private to a public company brings new opportunities, but also many new challenges — one of which is accurately forecasting sales. While there are many traditional ways forecasting has been accomplished in the past, Pluralsight has a strong desire to invest in new ways of doing things and looked to me to create an additional forecasting perspective using data science tools.
First, let’s go over the three main numbers for sales forecasting that are typically calculated; bookings, billings, and revenue. A sale is considered booked if an account executive in sales marks it as being sold. This typically happens after a sales contract is signed. It’s the first step in the income generation process. After a sale is booked, customer information is put into the company’s billing system and an invoice is sent to the customer. The sum total amount that is sent out at this step is the billings amount. This typically happens shortly after the booking, so many companies will only work with one number, either billings or bookings. After an invoice is created, the customer typically has 30 days to send in payment. The amount that is actually collected is revenue. In a contract SaaS based company like Pluralsight, you also only get to recognize revenue once you have delivered a service, which means most of revenue forecasting involves looking at which contracts are currently active and have already paid. Revenue for a quarter is the easiest to predict out of the three numbers because most customers will pay on time and you have a nice 30 day lag between when a bill is sent and revenue starts being recognized. Bookings, at the other end of the process, is the hardest to predict because there are very few leading indicators. A deal can be won very quickly, or drag on for years, and it’s hard to predict what’s going to happen ahead of time. At Pluralsight, I was asked to focus on producing a billings forecast.
Multiple Departments Make Their Own Forecast
Sales Driven Forecast
Traditionally, sales billings forecasting has been done in two departments: sales and finance. The sales forecasting approach involves each account executive making a prediction about how much they think they are going to make this quarter. Each sales regional manager then collects these numbers, adds them up, and makes their own adjustments to what their region will produce. These numbers are then rolled up to the next level of the management chain, combined and adjusted. The process continues until you get to the head of sales. It’s a very human intuition centric approach.
Finance Department Forecast
On the separate end of the spectrum is the forecast generated by the finance department. As with processes related to the finance department, this forecast revolves around an Excel spreadsheet. In this approach, multiple assumptions are made to create formulas that describe relationships within the company and eventually lead to forecast number. A typical formula might involve taking the number of expected account executives at the company, multiplying it by the expected close rate of past account executives, multiplying that by the expected deal size, and doing more multiplications and divisions based on assumptions until you get to your forecast number. These methods can get very complex and take input from HR, marketing, sales, and other parts of the company. While this is more numerically rigorous than the sales forecasting method mentioned previously, it still relies on a lot of human intuition about what the pertinent relationships are and how important each one is.
These two types of forecasts are found at almost every company, and they have served businesses well for decades. Data science also has a lot of forecasting related tools, so what can a data science forecast add that the other two methods lack? Three factors worth exploring are unswayability, longer forecasting time horizons, and automation.
Unswayability
One thing that you’ll notice with both of the above examples is that they rely heavily on human guess work. With the Sales Driven Forecast, each account executive and manager has an incentive to give a low forecast number. It’s much easier for a salesperson to raise their forecast number, but lowering it will typically result in leadership disappointment. Also, if the forecast is looking too low, upper sales leadership can exert influence on their subordinates to increase their forecasts, even if the individual account executives don’t feel the forecast should go up. Additionally, it is entirely possible for the sales force as a whole to become overconfident in their abilities and generate forecasts that are higher than the final results of the quarter will show.
On the financial modeling side, if a forecast is looking too low, a sales leader can ask for more resources based on what the model says. For example, in the spreadsheet made by finance, there is typically an assumption made in the relationship between marketing and sales, say for every dollar spent on marketing, we make a dollar and five cents in revenue. If leadership wants to increase its forecast, one way to do it is to increase the marketing budget. However, there is no validation work done to assure that the magnitude of the relationship between marketing and sales is accurate. This can cause overconfidence in the raised forecast values.
On the opposite end of this spectrum is a data science approach. Typically, data science models are built to remove human influence from the final output. For example, if a forecast model is purely based on a time series input, the only way to raise the forecast will be to do better in the future than what the model has predicted. It’s a great way to act as a check against over-optimistic or over-pessimistic thinking with the other finance and sales model forecasts.
Longer Time Horizons
The job of an account executive has a huge risk/reward trade off. Their work also has possibly the most objective success criteria of any modern business function. They either exceed their expected sales targets, and possibly get a pretty hefty bonus in the process, or they miss their target and may find themselves out of a job shortly thereafter. Because of this, sales teams can get myopic and tend to focus almost exclusively on the current quarter. There may be some idea of what next quarter is looking like in terms of pipeline, but anything beyond that horizon is of minor concern. Therefore, getting an accurate prediction from sales for any horizon beyond three months is very difficult.
Conversely, time series predictions can be extended out for however long you care to. Of course, extending a forecast farther out increases the error rate and uncertainty of the prediction. How far you can extend your forecast largely depends on the data that you are building your model with. Luckily, with back testing, you can get a quantifiable estimate of how much you can expect the forecast to be off by in the future. At Pluralsight, we’ve been able to create reliable forecasts up to 9 months into the future that are typically within 15% of actual billings. At the start of a quarter, or a three month horizon, our performance is typically in the 5%-10% range in accuracy. For comparison, the sales hierarchy generated forecast mentioned earlier is closer to 20% accurate with a three month horizon. The significance of this difference in accuracy for a company our size results in tens of millions of dollars over the course of a year and can mean the difference between meeting or missing Wall Street’s expectations.
Automation
It takes a lot of time and coordination to get all of the sales numbers together from every account executive and manager. It also takes a considerable amount of time to get finance to update their financial spreadsheets. However, once an adequate data pipeline is set up for a time series forecast, it can be run automatically. At Pluralsight, the traditional forecast models mentioned above are typically updated once every two weeks. However, the data science forecasting model our team manages updates daily and does so without human intervention. This is great for the data science team because it frees up their time to work on other projects. It also gives leadership a more on-demand look into what the current outlook is. If a leader needs to know what the forecast currently is, they won’t need to wonder about the timeliness of the guidance.
Implementation
Now that we’ve laid the groundwork to understand why a data science approach to forecasting has value for a company’s forecasting approach, we need to actually build it. However, time series work, which is what this type of forecasting falls into, is very different from most traditional data science work. Many data science and machine learning problems leverage datasets with hundreds of columns and tens of thousands of rows. When working with time series, you only get two columns of data describing each sample i.e., the date/time the sample was collected and what the value is. This makes many ML approaches inappropriate to use right out of the box. Also, your data will be much more limited. While a structured classification problem may give you over 10,000 rows to work with, a time series with daily resolution will only give you 365 values for each year of data that you’ve recorded.
While there is a lot of training material out there to teach yourself classic supervised and unsupervised machine learning algorithms, there isn’t as much when it comes to time series work, which means that most data scientists will need to teach themselves this field if they haven’t worked on it before.
This is the position I found myself in when I was handed my first forecasting task. As mentioned at the beginning of this post, Pluralsight was preparing to go through an IPO and needed to improve its forecasting ability, so they came to me. They needed a forecast generated with a tight deadline of around a month since their current forecasting methods were not performing as expected. I had never done forecasting or time series work before, so I started to do research.
The AutoARIMA Letdown
After reading a bit, I inevitably came across the popular ARIMA model. Even better, there’s a package called Auto ARIMA that’s supposed to figure out most of the tuning for you. Sounds like a great place to start out to get a first iteration to build off of. I dropped in my data and got something that looks like this where the black line is the historical training data and the blue line is what the model predicts will happen in the future.
Not exactly the glowing forecast that you would expect given how much you can read about ARIMA models. If you are using ARIMA for the first time and see a result like this, the first thing you might do is consult the documentation and see what parameters you can adjust. However, what you see written down seems less than helpful. Here’s a few examples from rdocumentation.com
max.p Maximum value of p
Not exactly helpful. What is p anyway? Also, if you look forward a bit more you’ll also see a different parameter you can adjust
max.P Maximum value of P
Wait, so there’s two different “p”s? A lower case one and an upper case one? What makes these different?
As you look around the rest of the documentation, you see more of the same. Lots of jargon and little explanation.
If you’ve worked with ARIMA before, you know how to work with this and make adjustments. If it’s your first time approaching an ARIMA, this documentation is very frustrating. You’re going to need to study even more before you can even get your first model going. You look up the author of the Auto ARIMA package and find out that the guy has a whole textbook written on the subject “Forecasting: Principles and Practice”, which is available for free online. The book is fantastic, if you have the time to read and study it, but we’re on a tight timeline here. A company’s IPO deadline is looming and there’s C-suite executives waiting for a new forecast as soon as possible.
It’s this exact situation that a few data science researchers at Facebook identified when they came up with the Prophet package. They recorded this in a talk that’s worth watching.
These data scientists identified this pain point and made a forecasting package that will produce decently accurate results in most situations on the first run. As they describe in the abstract to their research paper:
Forecasting is a common data science task that helps organizations with capacity planning, goal setting, and anomaly detection. Despite its importance, there are serious challenges associated with producing reliable and high quality forecasts especially when there are a variety of time series and analysts with expertise in time-series modeling are relatively rare. To address these challenges, we describe a practical approach to forecasting “at scale” that combines configurable models with analyst-in-the-loop performance analysis. We propose a modular regression model with interpretable parameters that can be intuitively adjusted by analysts with domain knowledge about the time series. We describe performance analyses to compare and evaluate forecasting procedures, and automatically flag forecasts for manual review and adjustment. Tools that help analysts to use their expertise most effectively enable reliable, practical forecasting of business time series.
Additionally, the documentation is also straightforward and much easier to understand for many new data scientist practitioners. There are also a lot of built in tools for diagnostic work and back-testing to make sure that your forecast has a good chance of being accurate. It’s for these reasons that a lot of our forecasting models initially started to be built on this platform.
Next Steps
Now that we have a good background understanding of what a revenue forecast entails and know what tools can be used, it’s time to actually build it. Prophet is great at helping you build a decent forecast quickly, but Prophet is built to deliver a daily forecast at daily granularity, but a revenue forecast needs to be created at the quarter granularity. We need to go over appropriate metrics to know if the forecast is accurate or not. We will also go over additional regressors that can be used to make the forecast more accurate. Stay tuned for the next series in this blog to learn more.
Have you worked with forecasting before? How did it go? What algorithms did you use? Was it received well? Share your comments below and leave an applause if you enjoyed reading.
