How Does the Lean Startup Impact the Chances of Startup Success?
There are plenty of cases that illustrate how the Lean Startup methodology has helped successful startups in their first steps: Intuit, zipCar, Dropbox, IMVU, Wealthfront or Grockit are just a first brief list. Since Eric Ries published The Lean Startup, the main ideas contained in it such as minimum viable product or pivot have spread like fire across every group of stakeholders concerned with innovation and entrepreneurship, from actual entrepreneurs to venture capitalists through incubators, academics and, specially, corporate innovation departments. The method is so clear, logical and even intuitive in the way it frames new venture ideas as hypotheses to be validated, that many could in fact ask how did we even ever manage entrepreneurship in any other way that wasn’t Lean.
Nevertheless, seven years after the book was published — and ten after the coining of the term — it is strikingly hard to find any statistical evidence that the methodology increases the chances of success of those who use it. It’s true that this is a problem with many management tools and methodologies — as promising as they are they are rarely statistically tested to whether they truly help accomplish the goals they were designed for. As a community we have to be aware of the fact that the methodology helped the startups mentioned in the previous paragraph achieve success doesn’t necessarily mean that it does so always, and perhaps not even in a majority of cases. By focusing solely on a series of selected success cases that illustrate the Lean Startup’s contribution to success we are likely missing the whole picture and falling prey of an availability bias to defend our own passion for a method that promises to increase our chances of success as entrepreneurs.
It is only fair that we look at the Lean Startup methodology as a hypothesis of a methodology to help startups succeed.
In fact, it is only fair that we look at the Lean Startup methodology as a hypothesis of a methodology to help startups succeed. In the past years we have built the method — far beyond a minimum viable product — tried it with countless startups and perhaps now, although a little late, we should take a step back to measure and learn by looking at the big picture to inquire into its track record: are startups applying the Lean Startup methodology more or less likely to succeed than startups that don’t? Can we predict success based on the lean mindset of a certain team? In summary, does the application of Lean Startup principles by a team of entrepreneurs cause an increase in their probabilities of success compared to those that don’t?
Are startups applying the Lean Startup methodology more or less likely to succeed than startups that don’t?
Of course there are countless factors impacting the chances of success of a startup: timing, team, network, location, access to funding, actual funding and market are just a few. Nonetheless, if the Lean Startup truly helps make startups more successful it should show in the total aggregate data.
There seems to be only one study looking at the question in detail using statistics. It is a dissertation written by Gaute Terland Nilsen and Nicolai Arguillere Ramm during their MSc in Innovation and Entrepreneurship at the University of Oslo titled “Lean Startup: A Success Factor?”. In the study, the students plotted 47 startups to compare their use of Lean Startup with a Success Score.
At first glance the result certainly looks disappointing for Lean Startup fans. Where most expected to see a straight line (or even perhaps a curved exponential one) from bottom left to upper right indicating that the more a team of entrepreneurs uses the Lean Startup the more successful their business is, the plot shows a rather scattered 47 points in the matrix, showing no significant correlation between the use of the methodology and the success of the teams and businesses using it.
But a second look at the scatter leads to another insight. Imagine that we divided the quadrant of the plot into two triangles by drawing a line from the lower left corner to the upper right corner; then, the upper left triangle would barely have any points representing each of the 47 startups plotted, whereas the lower right triangle would include practically all of them. In other words, although many startups using the methodology fail, there is barely a single startup that doesn’t use it that succeeds. Thus, using the Lean Startup is no guarantee of success, but not using it is almost a guarantee of failure.
Using the Lean Startup is no guarantee of success. Not using it is almost a guarantee of failure.
To this insight we must add a caveat indicated by the authors regarding the way in which each startups’ use of the methodology has been quantified: “informants that say they are familiar with Lean Startup and believe they use it in practice, actually don’t know what the framework is about — and therefore do not use it in practice […] informants who know about Lean Startup choose not to use it in practice, or only practice some aspects of the framework and not others […] many of the informants who are not familiar with the Lean Startup framework are using it without being aware of it”.
Thus, if an entrepreneur is not familiar with the Lean Startup framework but uses it, this entrepreneur would be marked as not using the approach in Nilsen and Arguillere’s study. Some of the points in the plot may be considered as “not using Lean Startup” albeit actually applying it unconsciously. The fact is that Lean Startup is the theoretical formalisation of an approach to problems and project management that many entrepreneurs carried intuitively before it was a methodology at all. As the authors point the same applies today, where there may be entrepreneurs applying it intuitively regardless of their awareness of its existence.
This leads us to a follow-up question: how can we discern whether entrepreneurs and innovators have the right mindset, what we could call a lean mindset when approaching ideas and problems, beyond their theoretical knowledge of the method? If we could define and, to a certain extent, quantify this mindset, we will be much closer to evaluating its impact on the potential success outcomes of a startup team. If we can quantify this mindset and it shows a correlation with a team’s probabilities of success we will have an immensely powerful tool to help guide a wide range of investments, from training at corporates and all the way to seed investment decisions or the choice of CEOs at Venture Builders or corporate innovation projects.
How can we discern whether entrepreneurs and innovators have the right mindset, what we could call a lean mindset when approaching ideas and problems, beyond their theoretical knowledge of the method?
We will follow up on this in the coming months.
Reference: TERLAND, Gaute N. RAMM, Nicolay A (2015) Lean Startup: A Success Factor? A quantitive study of how use of the Lean Startup framework affects the success of Norwegian high-tech startups University of Oslo. Oslo.