Crafting Products. Knowing the quality of what you’re building.

In the fast-paced world of technology, it is crucial to ensure that the products we create maintain high quality. This article aims to provide an overview of how we can measure a product’s quality, from understanding what quality means to the various strategies we can use to evaluate and improve it.

But… What’s that all about?

First of all, what defines a quality product? Or what do we mean when we talk about the quality of a product?

Generally, we perceive a product as high-quality if it efficiently meets our expectations at all times. For instance, considering a jar of tomato sauce, we regard it as high-quality if it satisfies our taste, aligns with our health standards, and has a long shelf life.

The criteria for technological products aren’t that different. Ultimately, we want our application to function as expected, be user-friendly, avoid causing unexpected issues, and always be available. We can consider it a quality product if it meets all these requirements.

And, as always, everything depends on the context. When I joined my current team a couple of years ago, our product was giving service to 10 stores. The purpose of the application was very clear: solve new problems and add new features as we were discovering the needs of our users. But with time, we started expanding, and now we are providing service to over 1700 stores. Now, it is not just about adding features but also ensuring that operations run smoothly, the application is scalable and can endure sustainable growth in users as well as features.

So it’s really important to identify the purpose of what you’re building in your specific context. In Mercadona Tech, we say that if something is not helping to “sell lettuce,” then it is just a waste of time.

And how do we measure it?

Metrics, metrics and more metrics

Metrics will tell us user behavior and help us determine whether the application is really being used as expected or if it falls short.

Do users understand the new functionality? Are they using it? Does the new feature solve the need that was initially identified?

Metrics should be able to answer all these questions (and many more), and their answers will indicate whether we are on the right track. This is where it is crucial to have a reliable system for tracking user feedback and a high level of maturity in terms of system observability to help you find out if something is not going as expected.

Get down in the mud and ask users

At Mercadona Tech, we are fortunate to be able to go to a store or warehouse and ask users directly. That gives us first-hand insights into how our products fit in and validate if they are really meeting our expectations.

But that’s not always the case, so you must enable a channel where users can give feedback. In this way, we will be able to gather all these inputs, identify our users’ concerns, determine if there are patterns that repeat themselves, etc.

Another indicator of a quality product is being able to do instant onboarding, where anyone familiar with the task knows how to use the tool. Just because it is an internal/logistic product, it doesn’t have to come with an instruction manual.

One of the KPIs used by our teams is the number of incidents/problems that our product records. This way, the whole team knows that the quality of our product is one of our top priorities.

How much time does it take to add a new feature?

We all want to work on new projects. It’s like working on a blank canvas, where decisions are not limited by previous work, and we can implement things a little bit as desired. Changes are introduced at an incredible speed, and everything goes smoothly.

Over time, without us noticing, what used to take a few hours now takes days and then turns into weeks. What used to be a simple change in functionality has now become an arduous job. Making adjustments to code on one side unexpectedly impacts other areas. Pipelines that used to run in minutes now take hours.

If we neglect good practices, we risk falling into a constant maintenance cycle without the ability to innovate or introduce new elements. Maintaining an application for 10 stores is not the same as maintaining an entire supply chain. This is a clear example of how, due to the product’s organic growth, our code has deteriorated to the point where it has become nearly unmanageable.

And even if you are careful with the product’s quality, some costs are inevitable and keep growing over time. This concept is known as basal cost. Here is a post by Eduardo Ferro (in Spanish) explaining it in detail. But don’t be mistaken. Taking good care of your product is essential to delay an unmanageable big ball of mud as much as possible.

Going back to my team example, when working for just 10 stores, we could add features quite easily. The codebase was small, the application’s load was reduced, and so on. But now, serving many more stores, we have to take into account other things when adding features: Is the database going to hold this? Are we going to be able to hold all that traffic? Do we have alerts that would warn us in case of an undesired behavior? How does this new feature interact with the existing one? And I could go on and on…

The feared incidents

Whenever we introduce something new, are we inadvertently breaking both existing and new functionalities in the process?

On one hand, incidents can have a technical root cause. If every attempt to introduce a new feature breaks something, it clearly indicates that our code lacks the necessary quality to keep pace with our development goals. There are also the old bugs that have been around longer than we have been in the company, but only palliative measures are taken and not corrective ones, often due to “lack of time.”

On the other hand, we may encounter user incidents related to our product, where users have difficulty understanding how to use it, leading to frequent errors.

And things get exciting when you start scaling. A feature can work wonders when used just by a few stores. We have a group called “laboratory,” these stores are used to undergoing constant changes, so they know how to handle unforeseen events. On a technical level, the complexity and amount of data we hold in this environment are limited.

But of course, it is quite another thing to launch that same functionality to the rest of the supply chain, where there are many more users with different profiles, the system has to withstand significant traffic and corner cases that are hard to predict. Some things “get through” lab, so we still need to keep iterating the solution once we get broader feedback.

Ok then… What do we do now?

First and foremost, it’s essential to identify the factors that have led to this situation. Typically, it’s not just one issue but rather an accumulation of various factors over time.

Knowing whether we comply with the quality standards required in our context is a constant exercise. It’s insufficient to conduct a checkpoint at a single moment and never revisit the topic.

Every time we have introduced a new feature into our systems, we should ask ourselves what difficulties we have encountered in doing so:

• Did it take longer than it should have because the code was not understood, or was it too tightly coupled?
• Does our CI/CD system take too long to deploy our changes, so it takes too long to validate them?
• Are many users getting in touch with us because they are confused?
• Is our system scalable and can it endure sustainable growth in users and features, or is it constantly short on resources?
• Are we frequently encountering corner cases that were not initially considered?
• Most importantly, does the change have the desired outcome for the problem at hand?

Identify if we are at a previous point and evaluate if it is necessary to stop and reevaluate our priorities before introducing new things.

Let’s wrap up

Ongoing journey that demands continuous evaluation and improvement:

1. Context Matters: The product’s purpose and context dictate its quality definition. As our product expanded from serving 10 to over 1700 stores, our focus shifted from adding features to ensuring scalability, operational smoothness, and sustained growth.
2. Metrics and User Feedback: Metrics play a crucial role in understanding user behavior and validating whether the product meets expectations. Whether directly obtained or through channels, user feedback provides valuable insights into product usability and effectiveness.
3. Balancing Speed and Quality: The speed at which we introduce new features should not compromise product quality. Monitoring the time required to implement changes, handling incidents, and maintaining scalability are essential aspects of balancing innovation and stability.
4. Continuous Evaluation: Regular checkpoints and retrospectives are essential to assess whether quality standards are being upheld. Identifying underlying factors contributing to issues and reevaluating priorities are key steps toward maintaining product excellence.
5. Learning from Challenges: Incidents and difficulties encountered during feature deployment offer valuable lessons for improving development practices. Understanding root causes and addressing fundamental issues ensures sustainable growth and innovation.
6. Proactive Maintenance: Investing in proactive maintenance and addressing technical debt prevents the accumulation of challenges that can impede future development and innovation.

By prioritizing these principles, we can effectively navigate the complexities of product development in a dynamic technological landscape.