Continuous deployment. How to enhance the quality of our products.

Introduction

Hello there! I’m a backend developer on the Stocks team at Mercadona Tech.

Today, I want to discuss continuous deployment and its caveats. My goal with this article is to alleviate the natural apprehension associated with deploying in production. By providing reasons and data, I aim to demonstrate the potential impact on our products that can have this technique.

The mantra is the more frequent and faster, the better.

So, what is it?

OK, but first things first. What is continuous deployment? To keep it really simple:

It is a software development practice where code changes are automatically and continuously deployed to production environments without manual intervention.

While the concept is straightforward, it requires substantial work behind the scenes. So, why should we bother? What benefits does this approach offer? Here are a few reasons.

OK, but why?

The deception of staging environments

We’ve all been there. We’ve worked (or are working) in companies where we must pass our developments through specific pre-production environments to ensure they are high quality for deployment.

These environments exist to assure us that everything will go as planned and we won’t cause any issues in production. As part of this, we spend significant time setting up production infrastructure replicas, making daily copies of the production data to these environments to mirror them as closely as possible, and doing many other tasks.

However, experience has taught me that these environments don’t guarantee a smooth deployment. No matter how closely we try to mimic the production environment, they will never be an exact replica. And these differences tend to be a key factor.

It’s important to remember that users are unpredictable. Our app will likely be used in ways we haven’t anticipated and couldn’t possibly replicate in non-productive environment.

💡 Rather than being helpful, non-productive environments can often become an obstacle in the development cycle.

To reduce incidents

A common concern when deploying to production is the risk of causing incidents. Thus, the initial impulse is often to deploy as infrequently as possible, assuming this will minimize the potential impact on the production environment.

But let’s step back for a moment and take another look. The longer we postpone the release, the more features will gather in that release, increasing the probability of issues appearing in production.

If we opt for smaller deployments, with only one feature or even parts of a feature, the likelihood of causing an incident is significantly reduced due to its minimal impact on other parts of the product.

💡 The later we make a release, the greater the risk of creating a high-impact incident.

To detect issues faster

Keeping with the long releases issue, it will be difficult to identify and fix bugs in our application if we release updates every three months. Furthermore, recalling the reasoning behind specific programming decisions made months ago, which may have led to the problem, can be challenging

On the other hand, consider the scenario where we deploy once a day. If we detect a problem in our product, our main suspect would be the previous day’s release. Since this release contains a limited amount of code, it’s much easier to identify the problem.

With that, we will be able to reduce the impact of the incident dramatically.

💡 The larger the release, the more difficult it is to pinpoint the source of an issue and the greater the impact on our business.

To add value

To truly determine the value of a new feature in your product, the best approach is to put it into production. We can assess its true worth by observing how it impacts our product and how users interact with it.

In his book “Extreme Programming Explained,” Kent Beck emphasized the importance of minimizing feedback loops for an XP team to operate effectively.

💡 The longer it takes to reach production, the longer the feedback loop becomes, and the greater the risk of wasting our time and money.

How to do it

However, it is important not to make this decision hastily. Deploying without proper preparation can be negligent, resulting in serious problems that could quickly lead to failure.

Continuous deployment is just the tip of the iceberg. Behind it, several requirements must be met to implement it successfully:

• A continuous integration system, where our code is automatically validated and integrated into our repo.
• A good level of observability in our product and systems allows us know the state of our application at any time (you can check the article I wrote about it).
• Tools such as Feature Flags (you can check the article by my colleague Felipe) to be able to react easily to a problem.
• A culturally mature team (XP, TDD, pair programming, etc.)

Ok… But what about us

Here is the lead time of my team during last October.

That time to deploy is just the time it takes to build and deploy the new image once the pull request has been merged.

The time in the branch is partially “polluted” by the dependabot, as it creates the branches really early in the morning. But we are already going to merge those pull requests once created automatically

And here is the deployment distribution in the same month.

It looks like on Thursdays, we don’t code that much 🤔 (spoiler: typically is the retro + planning day). Usually, we use Fridays for workshops and self-training.

At Mercadona Tech, we are fortunate to have a Developer Experience team. They have provided us with tools that allow us to deploy to a small portion of the production environment (referred to as a “lab”) immediately after merging a pull request. Furthermore, this release will automatically be deployed to the rest of the production environment 24 hours later.

Wrap up

Continuous deployment enables us to consistently modify and enhance our product in a rapidly evolving world. Therefore, establishing shorter feedback loops will allow us to:

• Decrease the number and impact of incidents.
• Minimize incident recovery time.
• Effectively enhance our product’s value.

So, how do you manage your deployments? I would appreciate hearing about your experiences in the comments.