Fail often and fail fast, but fail for a good lesson
The fallacy of the failure mantra and how to efficiently drive your experiments toward your goals
“Failure is good. Fail often and fail fast”. This mantra is often used in the startup ecosystem to stimulate experimentation and scale learning. The idea behind it is that failure generates important lessons and experiences required to keep moving your company forward.
Fail often indicates you should always be testing new things to keep your learning curve going up. Fail fast indicates your experiments shouldn’t take long in order to minimize losses and quicker apply the learning into the business. Having an established and continuous experimentation cycle is a key part of the startup adaptability and growth course.
However, while the quote has a lot of truth, it also brings hard concerns. Consider my explanation above regarding the fail often and fail fast concepts. Neither indicate that failure is obligatory, although both highlight the importance to learn. Failure is just a possible consequence. In fact, embracing failure is misguided and should be avoided by all means. Your starting point can‘t’ be on the loss, but on the possibility of progress.
Following this perspective, one might say it’s impossible to succeed on everything you try, which is absolutely right. Circumstances beyond our control may come up. On the other hand, not every failure is worth the lesson that was learned, if there even was one. Bare in mind that each and every experiment you do takes time and money, not to mention the opportunity cost. Hence a good failure, in other words a failure that is worth the time and money spent on it, will only be good if: the lesson was truly meaningful for the business; the experiment you realized wasn’t done before.
On short, a good failure will only be possible if the resulted lesson is valuable and fresh. And for that, you require all the information you can get and to really think your experiment through. That’s why your experimentation process must be done in the most structured way possible.
Eric Ries, in his book “The Lean Startup”, presented an interesting approach that was quickly disseminated and adopted by entrepreneurs and startups. He proposed applying the scientific method, a common technique used mostly in scientific research, to business. It’s basically turning every action you take in product development, marketing or any other department, into an experiment. It entails: (i) creating an hypothesis; (ii) defining what success would look like in terms of metrics and goals; (iii) structuring an execution plan to test the proposed hypothesis; and (iv) observing and analyzing the results. If the outcomes don’t match the goals, at least you learned what not to do. But if they do match, you have just opened a new path to be explored.
“It’s about having a hypothesis, and testing it. If the results don’t match your hypothesis, you’ve got data. If the results do match your hypothesis, then you have a discovery.”
Rob Shelton, Global Innovation Chief of PwC.
In this regard, I’d like to introduce a 3 steps cycle that takes into consideration the scientific method and can help you establish a good experimentation methodology for you and your company:
- Incorporate past lessons to not repeat obvious mistakes.
- Plan your experiment to minimize risk.
- Execute like there’s no tomorrow.
Incorporating past lessons
Not every failure must be yours. Repeating mistakes that could be easily prevented is an awful way to spend resources. Do your homework and research about it before getting started. The lack of useful information can’t become the kiss of death of your promising idea.
There’s always an enormous chance someone already thought about something similar to what you are thinking and went ahead to test it. The truth is we shouldn’t only incorporate knowledge from inside our organization. Outside, there’s a myriad of testimonials, articles, posts and people that can aid you understand the problem you're facing and even handle you the answer you seek. I’ll show a few sources below.
First of all, if you are new to the startup environment and don’t even know where to begin, I strongly recommend reading Sam Altman’s Startup Playbook. He compiled several advices he gave entrepreneurs during his career as Y Combinator’s president and, tough he focus on what you need to know to start a startup, he also gives a nice overview on how a startup is structured. This can help you draw the guidelines for future experimentations.
An interesting source is Autopsy, a website that offers a collection of lessons from failed startups. Don’t take the same bad fortune those fellows had for yourself, learn from their mistakes! Similar to Autopsy, there’s also Insights.vc, where they selected several material from market gurus providing insights in different subjects.
In case you’re aiming to test a new marketing tool, you should definitely ran a comparative analysis of different tools available that attend your need. Take a look at Growth Verse. They developed an interactive map with several marketing tools categorized by the problem they solve: marketing automation, organic acquisition, analytics and much more. Pick one, check their top customers and how well backed they are. Also, check Maqtoob, an amazing and free tool discovery website that shows online ratings. It’s the kind of research you want to do before starting negotiations with a new provider.
Of course you can frequently ask other people in Quora. Quora is a Q&A website that pretty much became the nest of entrepreneurs, VCs and others looking for a piece of advice. Ask for an opinion on a new tool, a new channel or describe a problem you are experiencing. And don’t forget to search for an answer to your question before posting as there’s a big chance someone already answered what you are looking for.
Also in the Q&A field there’s Growth Hackers, which promotes discussions and AMAs on different topics. Their trending topics can even spark new ideas on what might work for your business.
Last, but not least, there’s a highly underestimated and cheap alternative: talking to people in startups close to you. You would be surprised how well people respond to others looking for advice, especially if you also have useful knowledge to share with them. Focus on startups whose business model is similar to yours, the conversation will be quite enriching.
Gathering pieces of related information from different sources will always be helpful when dealing with puzzles. This incorporation mindset shouldn’t be employed only for experiments, but for every problem we face. Actively seek people and places you can learn from. Look for it in Google, search on blog posts and articles, ask others. If all of that doesn’t solve your problem — that might happen when you realize you are facing something indeed innovative — it will still support the structuring of your experiment.
Planning your experiment
While repeating a mistake is bad, having to repeat an hypothesis because it wasn’t clear enough in first place or it was poorly carried out is even worst. Of course this is an issue that depends on several variables, but there’s something you can employ in every new supposition to help potentialize its success: to minimize risk.
“Risk comes from not knowing what you’re doing.”
Warren Buffett, American investor and CEO of Berkshire Hathaway.
Minimizing risk means going deep on the proposed hypothesis. From all the uncertainties lurking around ready to mess your plan, there’re questions you got ask yourself in order to uncover some of them. By going deep, you can clear the path so the only uncertainties that remain are the ones really worth the test.
One of the main reasons why we fail is due to not setting the right goals. Every hypothesis should consider 2 things: value and growth. The value hypothesis consists on improving value to the customer: creating a product or service people desperately need, improving your website UX 0r adding new features customers asked for. The growth hypothesis is about optimizing the growth of your company: choosing new marketing channels, developing a B2B department for corporative sales or testing the service of cheaper providers to reduce costs.
Increase the perceived value of the customer and boost growth are obviously related and, ideally, your hypothesis should comprehend both, although that won’t be possible all the time. For example, adding value to your customer’s experience will potentially increase your conversion and retention, which is fundamental for your company’s growth. In case your hypothesis doesn’t relate to neither, then it’s not important and shouldn’t be prioritized.
Next, ask yourself what success looks like. Is it an increase in sessions? Conversion optimization? A decrease in your CAC? Maybe reducing churn? By how much? Defining the right metrics is key here. A must read article from Andreessen Horowitz team briefly describes 16 fundamental startup’s metrics. Good hypothesis generally focus on fundamental metrics of the company and it is up to you to define them based on your business model and momentum.
When you start thinking about the metrics, don’t forget about simplicity. Your test shouldn’t include more than 3 metrics or it starts getting too complex to monitor and analyze. Remember you want fast and simple results to quickly employ the lesson learned in your company. If your hypothesis depends on too many metrics, consider breaking it down in different tests.
Establishing a halt is an interesting idea as well. You want to limit the amount of money or time spent, specially if you have more upcoming tests in your pipeline. This limit should work more like a guidance than a constraint. If you realize you still need one or two weeks to gather enough results to analyze and the trend seems good, keep going. Otherwise, if the ship starts to sink, jump.
After setting your goals and metrics, you need to devise a good action plan. A good plan includes all the necessary information you need to cover the majority of possible outcomes (incorporation step + risk minimization) plus a good division for each activity necessary to reach your goal (so your execution can run smoothly).
One of the best methods by its simplicity, in my opinion, is “Who Will Do What By When?”. It’s basically: defining an activity, assigning it to the responsible person and establishing a deadline. If that person is you, that’s even easier. You should also add a question of capital employed to limit the money spent in each step of the plan and avoid a team member overspending your entire budget.
Another good practice is second level thinking. Second level thinking is to think not just about the actions or reactions but the reactions to the reactions. By reframing the problem and trying to figure out the wildest scenarios, you can prepare for all the extreme outcomes. Until you study the extreme positions, you can’t know if you are ready for the worst.
Execution, execution, execution!
Planning without execution is wasted effort and can’t be the reason to an unsuccessful test. Executing your plan is supposed to be the easiest part of the entire experiment. You should be paranoid about it, paying close attention to details and really doing your best on each step of its course.
If your hypothesis ends up failing because it was wrongly executed, you have a big problem on your hands. Time and money were spent and you can’t even ascertain a learning. Did your hypothesis failed because its proposal didn’t match the business or because something in the middle went wrong?
Your goal is to fully and timely execute your action plan. Good leadership is key here to motivate the team by placing belief on the possible outcomes and by constantly following-up on the action’s execution. The team needs to feel their experiments are truly important and be accountable for the results of their work. Set dates for progress review (weekly is preferable) and a tracking template which includes all the metrics and activities.
“To lead means to achieve goals consistently, with the team, and doing things right.”
Vicente Falconi, Brazilian business consultant and CEO of Falconi Consultants.
An interesting technique for recurring status report is the 3 Ps: Progress, Plan and Problems. It includes discussing these 3 dimensions on your regular meetings. Progress means you can check all the latest achievements of the actions’ execution to analyze if they are in line with the proposed goals and deadlines. In the plan, you can have a good overview of the next steps (which should become progress in the following week). Finally, the problems are the challenges currently in the way of your experiment. Weekly meetings with the team are great to throw these challenges on the table and instigate brainstormings to come up with solutions.
PPPs works pretty well for both managers and team members. It gives the entire company visibility on everyone’s activities. Observable results spur action. If an employee sees his co-workers are fully executing and gathering good results and lessons from their experiments, he is encouraged to do the same. The more public your results, the more it triggers people to take action. Want to give the team accountability on the company’s results and get them discussing how to improve? Make it public. Make sure everyone is looking at those PPPs reports. Make sure everyone is kindled to execute at their finest.
Using the 3 steps cycle can help you run experiments more efficiently. And by being efficient, you guarantee that the resulting lessons are truly enriching for your business. It basically consists on: (i) gathering all relevant information and incorporating it in your test; (ii) planning your experiment by determining goals and metrics and developing an action plan; and (iii) guaranteeing the execution runs smoothly.
Structuring a good experimentation methodology is a competitive advantage. Unfortunately, the reality is that several startups are doing senseless roadmaps of experiments in order to reach pointless goals. They are so worried in moving fast that they throw anything in front of the customer claiming to be experimenting and learning from their reaction. You got to do things fast, isn’t it? Eventually, you will figure out what works, right?
Wrong. You got to ask the right questions before start seeking for answers. It’s important to move quickly, but it’s not wise to move purposelessly. I’m not asking you to lose several weeks formulating a plan only to realize it isn’t applicable anymore. Things change fast in a startup. I’m asking you to spend a day at max thinking your hypothesis through. It’s worth it. The output of knowing what to do is the speed and quality of good experiments and insightful lessons.
The learning curve of an organization is the turning point to its success. It helps nail something faster in order to scale it, gaining an edge over competitors and more credibility among investors. Rethink the way you experiment in order to leverage your learning. No one should ever set out to fail, but in case you do: fail for a good lesson.