Nowadays it’s more and more common to talk about continuous delivery. With the diffusion of Agile methodologies we could say that most of the companies are delivering code faster than ever before in computer history. This trend is amazing because it can give us the power not just to deliver faster, but also to fail faster, learn faster and therefore deliver real value to our customers faster.
To be able to manage this speed, our experimentation processes need to evolve at the same rate and more than ever we need to have tools and processes to measure everything, using these metrics to evaluate the impacts that every new feature or update causes in the product.
Despite that fact, this process of validation is frequently ignored or executed incorrectly — the latter is much worse due to the fact that you might come up with a wrong conclusion thinking you’re guided by the data, when the correct path might be completely in a different direction.
We need to be able to demystify some fallacies about product validation processes such as “This is just a small change, it wouldn't cause any changes for users”, or “We need to do this quickly, we don’t have time to validate”. We must stand against it and help everyone understand why we should have a different approach to actually validate everything we do.
In this article I intend to present the risks that such fallacies can bring and explain why they’re all misleading argumentations. I’ll also bring my vision about how to manage validation and an example so that you can get some reference to use in your work.
Risks of not validating
With a poor product validation, one of the first main risks is to be misled by your biased vision (or even with the wrong data) and take wrong decisions. These wrong decisions can build up a lot of side effects that might be disastrous to your company and ultimately to your users (which is also bad for your company).
Another side effect would be to waste time, your time, time of the engineers and everyone else involved with the development of the product. Time of your customer support team that eventually will have to deal with bad outcomes you might put into production. Time of your PR team to revert some crisis you might cause that goes south. Even when it seems cheaper to skip validation, on average you’ll probably pay a higher price for not validating.
If you don’t validate you won’t be sure about the value of your work. Overtime this can lead to demotivation, yours and over the team that’s developing the product. People usually aren't comfortable to work without getting any merit, and this can be worse if other teams are taking the measures and delivering results in the company.
Even if your work is delivering the desired results, one last risk is to affect other stakeholders or other areas of the company negatively. Imagine that you work at a News platform at a “reader experience” team, you change the structure of the news adding 2x more images and then find out that the readers love it and engage more. In the process of doing that, you drag most of the advertising spaces to the bottom of your page, killing the main source of revenue at your company. It’s important to consider all of the effects before making a change in your product, once again: you need to measure everything!
Common fallacies of people that don’t understand the need for validation
- We need to do this quickly, we don’t have time to validate. I ensure you that, on average, much more time will be spent if you don’t work on the necessary validation. Most of the time our hypothesis are wrong so the chance of reaching a worse state is huge, you’ll usually spent a lot of time working to fix it afterwards.
- We launched it and our KPIs remains stable compared to the past, we’re safe. I’ll give in to this point: depending on the size of your company, or the nature of the new feature this might be ok. But it needs to be done really carefully… Most of the times, especially if your company is big, you shouldn’t use this type of argumentation. There will be so many changes happening on your product at the same time you will never be certain that the effect of your changes will be isolated, also, external effects may also play a big role, like movements in the market (competition, clients, economy). It’s really dangerous to assume that, so the next time someone asks if it’s ok to validate a feature comparing past to present states the answer is usually no!
- Let’s make a phase rollout starting with a couple of cities and them expanding to the rest of the country. This solution may also work depending on the nature of the changes/products you want to validate. The heterogeneity of the places you want to compare might also be taken into account. If you work in a country such as Brazil like myself, where each state has very distinct demographics and culture you might be in a bad position to use this kind of validation.
- This is just a small change, it won’t cause any noticeable changes for users. So why the hell you should bother doing it? Everything has a purpose and every action has side effects. I saw this argumentation when teams are trying to change the tech behind their product, assuming that the product itself will remain the same, but this is almost never true. You need to measure everything!
- We didn’t have the tools to measure or execute the required validation so let’s ignore it. Once again it’s a matter of investing in the right things. You need to understand the value of proper validation/experimentation processes first. After you understand that you’ll see that it's a great idea to invest in a solution for that in the first place so that you can evolve your product faster in the right direction.
How can I validate my product or feature?
First of all, have clear hypotheses mapped: It’s very important to have strong hypotheses for every new feature or changes in your product along with a clear set of success metrics. It’s also important to map all of the potential risks making sure you can measure them and at last check out for any potential stakeholders that should be consulted — this last part is very important not only for the sake of alignment but also because these stakeholders might have a different perspective that can help you refine your hypotheses and risks (saving you time).
Always make small changes: If you change everything in your product at once, you’ll find yourself in a rough spot to understand what works and what doesn’t. Even if the overall results being better than the old version, maybe one of the new elements might be individually worst, you might end up being at a local optimum instead of the global one.
Make sure your team’s experiments didn’t conflict with each other: This problem is more common in larger companies, where a lot of teams might be working on the same screen or flux. In this case, it’s important to make sure that all of the ongoing experiments didn’t conflict with each other, the best here is to ensure that each variation is segmented in a specific portion of your users.
Example of a good validation processes
I’ll provide a past example from one of my teams here at OLX Brasil, responsible for the search for content in our platform (in case you don’t know, we’re an online classified).
Hypotheses: If we highly increase the quality of the top 20 search results, we might increase our buyers satisfaction and engagement with content on our platform.
Success Metrics: Decrease time buyers takes to contact listers during search; Increase volume of contacts; Increase number of overall users contacting listers; Increase number of closed deals in our platform. Increase overall CSAT.
Risks: Giving that one of our products is a boost for listers that might pay a small amount of money, if we stop to prioritizing this paid ads in exchange of better search results we can reduce its effectiveness, therefore the companies revenue; Another risk is to focus on high quality ads for a better search result, concentrating buyers contacts among fewer ads. If this effect happens we can reduce overall closed deals in the platform and start losing most of our listers.
Stakeholders: The teams responsible for paid products, buyer’s experience and the ones that works directly with liquidity management of our platform.
Experimentation process: We would made an A/B Test, for a percentage of our buyers we will show the search results just as before, and for the rest of them the new search order with 20 top quality results on top. We will measure everything necessary so that we can compare the results of both population of buyers in the terms of our success metrics, and also make sure we can measure all the risks. In the end, if the results are better and the potential losses (from the risks) are worth it compared to the benefits we’ll put it in production!
My dear reader, if I can leave a final message for you to take away from after all of these lines of text, I would say: Make sure you have the right hypotheses and that you can measure everything!
Validation is almost every time a must… if you are uncertain about whether you should do it, I ensure you it's something worth doing excessively
Make experiments, invest in tools for product validation such as A/B test tools. I’m definitely biased being an ex-data scientist and metrics addict, but trust me when I say it — and I made mistakes a lot of times during these years till I learned it- validation is something worthy of your energy!