Is the User Story done? It isn’t, if you don’t measure it!

I have worked for many years in the tech industry and, as a consequence, I have experienced different approaches to deliver features live. Some of these features were taking months before moving to production. In this short article, I will describe how we do it at Cazoo

During my past work experience, I have used the Scrum Framework, and each user story was following the most classic approach:

Where the definition of “done” consisted of a checklist containing items such as:

• Code is peer-reviewed
• Code is deployed to the test environment
• Code/feature passes tests
• Feature is approved by stakeholders

Only after all these steps were completed could the story be deployed to production. However, we were often collecting multiple user stories (i.e., “release”) before actually deploying the new features to production.

Since I joined Cazoo, my team has been using a different approach that I consider more powerful and meaningful: before we can tell that a user story is done, we need to measure the effectiveness of that user story in production.

Sprint process

During the sprint planning, we define how we can measure a user story and for how long.

If we cannot define these parameters, it usually means that the user story is not “valuable” and that we need to add some extra details to the user story.

Our sprint’s dashboard, therefore, consists of four statuses: “TO DO”, “IN DEVELOPMENT”, “MEASUREMENT” and “DONE”.

When we move the story to “IN DEVELOPMENT” we do a kick-off, where we define all the tasks necessary to complete the story. One of these tasks will be to create a new dashboard to monitor the effectiveness of the user story in production.

After making sure that everything has been tested properly and that the user story is meeting the acceptance criteria, we prepare our dashboard to monitor the user story and we deploy the new feature directly to production.

At this point, we can demonstrate the user story to the stakeholders. Most of the time, we can do it directly in the production environment.

If all the stakeholders are happy with the new feature, we move the story to “MEASUREMENT”.

Every morning before the daily meeting, the team has a 15-minute meeting dedicated to the monitoring. In this meeting, we look at all the dashboards that we have in place and we double-check that everything is working properly for the user story currently in “MEASUREMENT”.

After a specific time defined in the sprint planning, provided that what we have deployed did not cause any problem, we move the user story to “DONE”.

What if something goes wrong?

It is possible that the team notices that the work that they have done did not have the desired effect. In this case, we either create a new story, if there is some extra work to be done, or we move the story back to “IN DEVELOPMENT” in case we have to fix something that was not done properly.

Measure your assumptions

Sometimes we make assumptions regarding how to improve customer experience, although we are not sure about the outcome of our improvements beforehand.

For example, we decided to add a link to the different finance options available at Cazoo in order to help the customers understand the details of each. By doing so, we were also measuring how many times the customer had used the link, and thus we could decide whether to keep the link, move it to a different position or remove it, based on the customer usage.

The following dashboard represents the number of times the customer is using the finance options link:

It is obvious from this dashboard that the link is used regularly, and therefore our assumption about how to improve the customer experience proved to be correct.

Measurement matters

Being “Customer obsessed” and “Data driven” are two of the main values at Cazoo.

“In order to be customer-obsessed you have to be challenging yourself to make transparent decisions that benefit customers instead of the business” (from https://medium.com/cazoo/customer-obsessed-product-bb234f4334f2)

One step to achieve these values is to understand what is happening across the system, so that we can detect and resolve issues to keep our system efficient and reliable, and ultimately make our customers happy. As Peter Drucker said “If you can’t measure it, you can’t improve it.”