Has Lean Startup Methodology Changed Since I Worked at IMVU?

I recently did an AMA on Growthhackers.com and thought some of the answers were worth sharing as medium posts. I’ll also add a bit of content or elaborate on the answers.

A good question I got from Edward Stephens: How much has the original “lean methodology” implemented at IMVU had to change to keep up with current trends or has it and does it continue to withstand the tests of time?

My answer:

I joined IMVU just as the lean startup movement was starting outside of the company, so it was certainly an exciting and interesting time to be there. Some of the key things that I learned at IMVU that I still think are incredibly relevant:

• Start with a hypothesis and then come up with the simplest way to test that hypothesis that minimizes development effort to get maximum result. IMVU was the first company I’d worked at (this is back in 2009, so at the time was less common and prior to solutions like Optimizely) that had a fully built out internal A/B testing infrastructure that could essentially give the product management team the capability to create tests, turn them on/off, and then get a full suite of analytics down to regression analysis of key metrics within retention and usage to make key decisions about tests. This meant that the internal company culture was all about hypothesis-driven development creating the basis for many of the large improvements that we saw in the business.
• A product without customers is not a business. This isn’t necessarily surprising, but when you look at how product development has changed over the years, considering your growth and traction channels has become a key part of early product and business strategy and not something that the marketing team does after the fact. This was one of the key principles that Lean Startup had thought through by validating concepts with customers to ensure that there was actual product-market fit before spending a lot of money on acquiring users.
• Implement true continuous deployment. Another impressive part of IMVU was that they had a true continuous deployment framework at the company, meaning any code could be shipped at any time. While this is more commonplace now, at the time, it was revolutionary to be part of a system like that and realize how much flexibility it afforded when wanting to rapidly test new hypotheses with customers.

There are some things that I think people have taken out of context regarding Lean Startup. I don’t think Eric Ries (just my hypothesis based on hearing him speak a couple times) ever intended these to be the takeaways, but I think there are some things companies still get wrong about the framework.

• Companies forget to actually talk to their customers. Oftentimes, companies miss a lot of nuance in what they are shipping or interesting hypotheses because they skip the phase of actually showing concepts to customers or they don’t spend actual time with customers to understand why something worked or didn’t. Instead, they just look at the raw experiment data without actually questioning why or why not something performed the way it did. As far as I understand, this was never the intent of Lean Startup, but I still see a lot of companies that get this wrong.
• Companies forget to actually derive a hypothesis for their test. If you don’t have a good reason for why you’re running a test or you can’t easily explain it to someone, it’s actually worth the effort to figure out why a test matters and why you’re running it. Obviously, this applies differently now that it’s easy to test concepts such as layout, button color, etc., but every test has an opportunity cost, even if it’s not strictly coding cost as it used to be. Whether it’s the fact that you have to pay attention to the test, the fact that it takes 2–3 weeks for a smaller company to get to statistical significance before knowing a result, or a variety of other things, there’s a tangible cost to just “throwing tests at the wall” based on what has worked successfully for other companies without thinking about the context of your own organization.
• Companies misinterpret Build-Measure-Learn as an excuse to build almost anything without customer validation or hypothesis driven development. Very similar to the above two points, but I often see companies just building elements without thinking why they are building something. Build-Measure-Learn has a clear set of thinking steps that need to happen before building actually happens.

Product Hunt Podcast’s interview of Eric Ries also did a great job addressing how Lean Startup has evolved: https://www.producthunt.com/radio (maker stories episode 10 as you scroll through).