Success case: follow the goal, not the requirement
Several months ago we introduced OKR in our scrum-based process.
We build a personal cloud product made of a backend, an Android and an iOS app, a web portal and apps running on computers. Our business model is B2B2C, meaning that we sell the product to big companies, usually mobile operators, who integrate it into their systems, rebrand the apps and make them available to the end users.
Among the first set of Key Results associated with the Objective, we had to improve the rating in the app store for our iOS app.
A good rating is very important for several reasons directly tied to the business: the app becomes more visible, the rate of users installing the app improves, the activations increase, and so on.
One way to achieve this goal would be to work on the app quality, making it nicer and fixing some bugs, and indeed we did some improvements in these areas as well. However, we mostly focused on the rating process itself, which had to change also for compliance with some recent Apple guidelines.
The requirement
A comparison analysis with other apps was performed, and the input coming to the scrum team was to introduce a new interaction layer before the actual invitation-to-rate popup. In that layer, unhappy users would be invited to send a direct feedback, without going to the store. If this behaved as expected, happy users would give to the app 4 or 5 stars, thereby improving the rating.
In other times, the team would have worked as a feature factory, with the goal to build the requirement in a stable and beautiful way, as fast as possible.
This time, having a clear and explicit KR made the difference: the goal, or the expected outcome using an established terminology, was to increase the rating. In a problem solving mindset, the requirement was seen as a suggestion to get there, with the freedom to consider other paths. Definitely this change of approach brought more energy and engagement in the team.
Towards the sprint goal
During the sprint, UX and DEV people together actually worked on changing the flow (let’s call it the WHAT), introducing the ‘send feedback’ as required. While digging into the existing code, with the goal in mind, they came up proposing an orthogonal approach: why not also focus on the trigger that makes the flow show up in the user’s path (let’s call it the WHEN)?
In particular, the existing flow was presented after the successful completion of n actions in a row, counting any action. The team suggested instead to identify a few emotionally valuable actions, and present the flow upon the execution of one of those: it was likely that in such a moment the user was more willing to provide a good rating.
Going live
Once the changes were ready for being launched to get the first empirical results, we faced a showstopper: for bureaucratic reasons, our customers could not deploy the ‘send feedback’ as it was designed, a more complex implementation was needed.
We then decided to just quickly disable the WHAT, restoring the flow to how it was before, and send live the app with only the change to the WHEN as proposed by the team.
The results exceeded our expectations by a large margin. Here is the trend of the app store ratings for two of our main customers, monitored over a period of a few weeks:
Such a rapid improvement is the result of not only better ratings by each user, but also of a much higher number of users rating the app.
What’s next?
Having seen things going so well, we decided not to invest any more effort on the rating mechanism, except for applying the new logic to our Android app, which also got a boost in the Play store.
As of now, we keep monitoring the apps in the stores, and evaluating sprint by sprint if any of the new features that we introduce should be added to the list of triggers for the app rating invitation to appear.
Conclusions
As a result of this activity we could observe that keeping the team involved with the current company Objective and the related Key Results brings more motivation, better team work, and a higher quality in the decisions, as everyone can contribute with different perspectives; the price to pay, which — as a Product Owner — is to be able at a certain point to take a decision of what to do and, much more important, what not to do (by now), seems worth the benefit.
In this particular case, without the team’s input, the activity would have resulted in a complete waste of effort and time.
Introducing the OKR in our process is part of a larger initiative we have embraced to bring more and more the focus on outcomes: in other words, make sure that what we build has an impact on the behaviour of our customers or of our users, even if this might eventually mean building something different than what initially considered.
Focusing on the outcomes requires to release early and often, to be able to quickly learn what are the real effects of what gets introduced, to validate the hypothesis or not, and then to decide how to proceed in the next sprints. This is one of the challenges we are still dealing with, specially due to the business model we have, but where we are also experiencing much more openness from our customers.
Being able to measure is also fundamental: for this specific case the indicators could be easily retrieved from the stores, but we are investing a lot in defining, introducing and maintaining the KPIs on the backend and in the apps.
The other great takeaway from this activity, not process-related but more specific to our product, is that our app had been so underrated for so long, just because we were not raising the right question at the right moment.
Kudos to the entire team who built an app that is now finally appreciated for its value, and special kudos to who were mostly involved in this activity.
