Using Experimental Design to set sales strategy in a SaaS Startup

Tom Tunguz, partner at Redpoint Ventures and author of Winning with Data, recently wrote an insightful article titled Monte Carlo Simulations Of Inside And Outside Sales Teams In A SaaS Startup about the value of a balanced inside/outside sales approach.

Tom ran a simple Monte Carlo simulation to express in probabilistic terms the benefit of following a barbell portfolio approach a la Nassim Taleb: balance the high predictability / fast sales-cycle of your inside sales team with the slow sales-cycle / high upside of your field sales team.

From one math nerd to another, I loved Tom’s analytical approach to quantify what is qualitatively intuitive, and found myself wanting more.

Why stop at a Monte Carlo simulation, which provides a probability distribution of outcomes for a single scenario of assumptions? Why not conduct a multifactor analysis via Markov Chain or Experimental Design to understand which drivers are most impactful across multiple scenarios?

For a quasi-random process such as sales, Monte Carlo simulation is useful for helping you understand probable outcomes to make better choices (e.g. number of inside vs. outside sales reps).

Experimental Design is useful for helping you test which of those choices will give you the best return. (e.g. better to invest in closing faster or closing bigger?)

No matter the probability, I would rarely ever place a bet on the occurrence of a single scenario. The value to me of mathematical analysis is not the calculus of a single outcome, but understanding the sensitivities that impact the range of possible outcomes.

“An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem.” — John Tukey

I went through the assumptions in Tom Tunguz’s simulation:

• 3 months spent per lead for inside sales, maximum
• 5% month1 / 5% month2 / 10% month3 closing probability
• $50,000 ticket value per inside sales lead
• $750,000 annual quota per inside sales
• $3,000,000 booking capacity for inside sales (4 reps)
• 9 months spent per lead for outside sales, maximum
• 5% month6 / 5% month7 / 5% month8 / 5% month9 closing probability
• $500,000 ticket value per outside sales lead
• $1,000,000 annual quota per outside sales rep
• $3,000,000 booking capacity for outside sales (3 reps)

And back-calculated his assumptions for leads, that make closing probability and ticket value match quota:

• 6.98 new leads/month for each inside sales rep
• 1.82 new leads/month for each outside sales rep
• 6 qualified leads at a time that each inside or outside rep is able to carry, max

Holding constant the $750k inside / $1,000k outside annual quota per rep, I then added a number of new variables to explore the following what-if scenarios:

• What if the probability of closing in any given month was lower/higher than 5% or 10%?
• What if the revenue per lead was more/less than $50k/$500k?
• What if each rep received a higher/lower number of leads per month?
• What if reps spent more time per lead than 3 or 9 months?
• What if a rep could carry more/fewer leads than 6 at a time?
• How does the annual loaded cost of inside and outside sales reps impact profitability?

I created a 23-factor Definitive Screening Design, my personal favorite highly efficient experimental protocol from the brainiacs over at the SAS Institute, in order to assess the impact on:

• Annual Revenue (R2 = 0.77, probability > f-test = 0.0013)
• Annual Cost of Sales (R2 = 0.62, probability > f-test = 0.089)
• Cost of Sales as a % of Revenue (R2 = 0.83, probability > f-test = 0.0001)
• % of spend on inside vs. outside sales (R2 = 0.81, probability > f-test = 0.0002)
• % of revenue generated by inside vs. outside sales (R2 = 0.95, probability > f-test = 0.0001)

Interestingly, the probabilities of closing in each month were the least impactful factors. By far the most impactful driver was the number of outside leads per month.

Coming in 2nd and 3rd were the number of months spent by inside reps before dropping a lead, and the cost of an inside sales rep.

Fellow math-nerds out there will recognize that my engineering approach is akin to fitting a finite element mesh rather than going the more “elegant” route of solving a multi-order partial differential equation.

Computational power is so cheap these days: I challenge anyone to solve this PDE faster than I can write a script for my laptop’s Intel i5 processor to approximate the solution.

“All models are wrong, some are useful” — George Box

In comparison with Tom Tunguz’s Monte Carlo Simulation which predicts $5M in revenue with 60% from inside sales as the most probable scenario, my expanded simulation predicts $8.5M revenue with 45% from inside sales as the most probable outcome… a conclusion that is as useless as it is unlikely.

The value in this exercise is not prediction; it’s the sensitivity analysis that reveals which variables are most impactful at driving revenue.

Based on this analysis, from a purely mathematical perspective I would recommend for this business to:

• Ramp up marketing to drive lead generation, especially for the big clients
• Not let the inside sales team get too antsy; don’t be so quick to drop qualified leads
• Hire more sales reps! Don’t let individual reps get overloaded with too many leads
• Not overspend on sales engineers; closing a higher percentage of deals is less impactful than getting quality leads
• Be cautious about loading inside sales reps with too much variable compensation / do load up field sales reps with a progressive plan heavy in variable compensation
• Be flexible with pricing, especially if demand is elastic

There’s a sweet spot for this business if able to get 2–4 new outside leads per month (white-space in the contour profile above).

“In theory, there is no difference between theory and practice. In practice, there is.” — Yogi Berra

I know few CMOs or VPs of Sales who would do well by relinquishing strategic decisions to their CFOs. This model is purely mathematical and does not take into consideration exogenous factors such as macroeconomics, competition, demand cycles, or human emotion.

Experimental design is a powerful tool to complement, not replace, the strategic decision making process.

It’s important to remember that a model is only as good as its assumptions, and this example is specific to a business with the following sales assumptions:

- Low probability of closing (<10%)

- ARR of $50k for inside sales & $500k for field sales

- 3–4 months to close inside sales & 9–10months to close field sales

- 7 new leads/month for inside sales & 2 new leads/month for field sales

- Each rep capable to carry 6 qualified leads at a time

In other words, representative of a B2B software business selling a complex/niche application with strong product-market fit to both SMBs (inside) and enterprise (field) customers, and likely charging by the number of licensed users.

Data tells a compelling story; Monte Carlo Simulation showed the importance to a SaaS startup of balancing inside/outside sales.

Experimental Design enriches this conclusion by showing the value of thicker barbell plates: empower those field sales reps who you rely on to swing big to get more at bats, and those inside sales reps who you rely on for consistency to close with greater regularity.

This article was originally published on Oct 22, 2017.