Product Relaunch Journey
Learnings From a Failing Release
Failure is not necessarily a bad sign
Last week we launched a new Beta version of our app. This beta release brought an immense amount of negative feedback from our users, and I have collected some learnings that I think are worth sharing.
I’m writing this article on March 1st 2021. Let’s have a flashback to reach a better understanding of the story before we jump to the conclusion.
Since Q1 2020 we have been working on the idea of revamping our five-year-old app UX into a more modern experience that helps our users get informed reliably, while expanding our opportunities of showing different varieties of content, for different appetites and contexts. Since then we had multiple workshops, ideation, design iterations and user testing. The user testing result was positive, so we proceeded with the implementation in Q3 2020.
This UI change is a major project for us, we have tens of millions of active users and this change touches the core product that the whole company relies on to make money. So, it’s not a “build it and they will come” kind of challenge. We have to be very cautious not to break the business while building a cool product change. That’s why we wanted to lay out some foundational activities for testing out what we develop. It started with the user testing on click-dummy prototypes, and was followed by enabling thousands of our users to join a beta program -that I’m most proud of- , and it will continue with the gradual rollout until we successfully reach a rollout to 100% of our users.
Where are we now in this timeline?
Now we are at last week (25.02.2021), we built a major change to the UI introducing a new layout of our Top News section. The Images below show the old UI vs the new UI. We launched with hope that people will get the new UX and interactions on their own as they told us in the user tests we had earlier in 2020, so we didn’t invest in building an interaction tutorial or additional visual cues. And that was…. Well, we will see below what it was 😄
Back to today
It’s just a few days since we released this update to beta. Looking at the user feedback, I can confirm that they don’t get it on their own. Due to a technical limitation, that’s highly dependent on one of our partners, we changed the interaction to navigate between the Top News cards from swiping to tapping the left/right edges of the card. Internally we could figure this interaction out, or let’s say we didn’t receive many internal complaints about it, so we assumed it would go well with the users. 90% of the feedback we received was against this assumption.
User A: Why are you forcing me to read all of your cards?
User B: I can’t navigate the cards on my own. I will uninstall.
User C: This is a disaster!
And so on, almost all of the feedback going in that direction. Clearly, our assumption was incorrect, that’s a harsh learning, but how did we react?
Next day reaction
Hmmm, so, this interaction is not as intuitive as we thought. Hence, we decided to teach the users how it works. Developing something would not be as fast as launching an in-app message campaign, that informs you how you can interact with the app. This is not my favorite approach because the interface should be intuitive enough to help you interact without a tutorial, but given that we’ve a 5 year old UI, and we’ve changed the interaction model that our users are used to, then probably it’s not a bad idea to teach the user about the new interaction model.
We’ve managed to launch the campaign on the very next day and thankfully our users now are informed about this interaction. Is the problem solved? No, there’s still unfinished business here 😉
Looking at how these people are trying to interact, we figured out that they’re trying to swipe left/right. So, we decided to give the swiping gesture a second chance, and to invest in resolving the technical limitation which is highly dependent on one of our partners. Most probably before the end of the week we’ll have a better beta version ready, as we already exchanged some tickets from the sprint, that happen to have lower user value, with these high prio updates to be done.
Now to the main learnings
Along this journey, I’ve collected some valuable learnings that would be helpful to me, and fellow product managers and their product teams, who would go through a similar experience.
══ Invest In a BETA program
I can’t emphasize enough how useful it was to launch this beta program. Users are willing to get the latest and -hopefully- the greatest. These are early adopters who are willing to risk stability for the sake of getting the cutting edge updates. Once you have a beta program you unlock possibilities of fearlessly testing out new assumptions without sacrificing the business stability.
══ When in doubt, go with partial rollout
I was not fully convinced that users would get it on their own especially after we changed the interaction model from swiping to tapping. However, I decided with the team to see if it would work without the need to invest in building any tutorials or additional visual cues. Although we initially tried to make it clear via some basic visual cues like animations and progress indicators, many users still didn’t get it as clearly as we hoped. This resulted in having tens/hundreds of annoyed beta users who think that the app is taking control from them. So, we made some enemies for the new UI already 💩
If I have the opportunity to go back in time and change something about it, probably I’d just roll out to 10% or 20% of our beta users on the first day, to see the general feedback before rolling out to all beta users.
══ Invest in communication with the user
If we didn’t make it very easy for the users to provide beta feedback, we wouldn’t have figured out that the users are annoyed with the new interaction model. This investment required development, marketing, and product effort, but it pays off. It’s essential for closing the first-hand feedback loop between your user and your product.
Also, invest in integrating in-app marketing & engagement SDKs early. Without having our in-app messaging SDk integrated, we wouldn’t have been able to run the campaign teaching the users how to use this new UI, on the very next day.
══ Take user-testing results with a grain of salt
No matter how good the facilitated user testing was, the real test is when you hit the reality rock. I believe that facilitated user test results should be only indicative, the result should not be taken for granted and applied on a large scale in production without further validation on a small subset of the users.
══ Release early, release often
This is very important. Try as much as possible to ship value very soon to your users to validate your assumptions. I’ve really learned this the hard way, by working on products that never saw the light while putting a lot of passion and effort on them internally, but it was very dangerous because it’s very tempting to pile up assumption after assumption waiting for the perfect setup. Well, good enough is releasable, and releasing is your reality check. The feedback and KPIs you look into after releasing should guide your way.
Your product is ideal in your mind until you release. By releasing, you figure out what problems you need to focus on next, and whether you are already delivering the value you wanted or not.
══ Iterate
This is the fun part in software development, it’s easy for us to recover from our falls pretty quickly. We’ve had assumptions that were not correct, we’ve learned and came up with new assumptions that are worth validating. Our assumption was not correct, but it’s also pretty clear that it’s not the end of the world. I’m thankful that we’ve the luxury to fail and iterate. That’s how software development should work.
══ Don’t lose hope, and trust your team
Although that most of the feedback from the users is negative, I am absolutely grateful and proud of our team because:
- Without properly integrating this feedback access point and making it very easy for the users to provide feedback we would’ve acted very late. This feedback access point saved us a lot of negative ratings on play store, and made us closer to our users.
- It’s a validation that we’re shipping software to people who care. One of the worst things ever is working on software for numb users who don’t give a shit about whatever you change. Thankfully it’s not the case here
- I’m confident that we’ll recover from this pretty quickly, providing a more intuitive experience for our users
As a final note, remember that we’re in software, for most of us the world would not collapse if we try out new things and fail.
If we fear failing we’ll fear trying, and I’m pretty sure that a team that doesn’t try is not destined for success on the long run 😉
I hope you had fun reading this post, thanks for your time 🙏🏽!
PS: I am not a fan of locking such posts to users who have a paid Medium subscription, so I’m keeping it open for everyone. If you think this post was useful to you, it could be useful to others as well 😉, please pay it forward by sharing it to those people who might be interested.
