Photo by Gene Devine on Unsplash

Releasing With Control

As we all know, going to production can be anxious at times, and we all take a sigh before we create that new release, and stare at that deployment job to turn green — and if we don’t commit to the right deployment practices, we lose confidence and the anxiety factor grows as the days pass.

Fortunately, after a lot of market research on these problems which are covered in the book Accelerate, the following are two important ideas that have come to the table which can help gain more confidence when deploying to production

• Decreasing Cycle Time (Change Lead Time) — Time to implement, test, and deliver code for a feature (measured from first commit to deployment)
• Increasing Deployment Frequency — Number of deployments in a given duration of time

One key approach to achieving the aforementioned ideas is to push small changes. If we try to keep the push small (whenever possible), we can try to reduce the cycle time. With a reduced cycle time, we can prepare ourselves for production.

By pushing changes to production quickly, we create a space to receive feedback regarding what we released.

Instant feedback serves to reinforce knowledge by correcting mistakes, affirming competence, or debunking misconceptions on the topic — eduMe

And a beautiful cycle begins:

Now, let us look at practices on how we could achieve what’s mentioned above:

1. Trunk-Based Development

Trunk-based development helps us to introduce smaller changes quicker

• As soon as the code hits the main branch, it should be deployable.
• pushing to themain branch makes your code available to other developers, which means there are no long-lived stale feature branches. We don’t want long-lived stale feature branches because they add to the anxiety factor of breaking production.

2. Continuous Integration / Continuous Deployment

Coupled with trunk-based development, we aim to increase the deployment frequency by deploying as soon as there is new code.

• Every merge to the main branch should trigger running tests on the branch,
• If they all pass, they should be deployed to production

At first, one may think of the following concerns

1. Merging to main deploys to production? 🤔 It should at least be deployed at a staging or pre-production environment first where we can test it properly.
2. We want this feature to be demo-ed first and gather feedback from the team and the parties involved, where do we do that? We can’t just do that in production.

Back to Reality

These are important questions — let's keep them afloat on the side, and first understand how the application and deployment pipelines for the WEB are currently set up.

In a nutshell:

Application:

1. The Backend sends components to the front-end, and the Frontend then builds the page for the user. (XPA)

Pipelines:

1. Merging in main deploys to staging and pre-production
2. Creating a release tag deploys to production.

Now, Let us assume the following scenario given the current setup:

Scenario:
We implement the new feature, say X, and merge it to the main branch, a pipeline runs which deploys feature X on the staging and pre-production environments. All those who have access to the staging/pre-production environment can now test and play around with feature X to provide feedback.

Drawbacks:

• It is possible that the application does not behave identically across production and staging (or pre-production) environments. See more
• Since the new feature X has already made its way to the main branch, releasing the main branch would cause the new feature X to be rolled out to production. If any Feature Y needs to go production, it must wait until Feature X in the main is tested and ready to go, or Feature X gets reverted.

Releasing, the better way

Testing in Production

Pushing the feature to production allows us to see and experience how the feature behaves in the real world — but with that said, we cannot just release everything to production just to see if it works.

Feature Gates

To overcome this problem, we use Feature Gates via Statsig — they allow us to control WHO sees the feature, and WHEN they see the feature — allowing us to safely push our code to production and test directly in production with real data and real traffic.

Trunk Based Development

By using feature gates, we can easily release Feature Y because the visibility of Feature X is controlled by the Feature Gate, not by it being present in the main branch.

Thus, we allow ourselves to reduce the cycle time by going into production, gathering feedback, and re-iterating feedback quicker.

A Feature Timeline

Enough talk. Let's go through a real-world example:

Recently, we deployed the embedded video player that you can see on the article page on the OneFootball website, and we followed the same process called “Dog Fooding” to finally release the best and improved version of the feature to our users.

Here is how it went down:

1. Deploying the feature to production — but only available to the Consumer Web Team

We achieve this by configuring a section for the Consumer Web Team

And then wiring up this section with the feature gate that we created for the embedded video player.

2. Gather feedback from the team

Please witness the professionalism exhibited:

this thread is very long, but you get the point.

3. Deploy the Revised Version and Increase the Audience

Deploying the revised feature to production — availably to all employees at OneFootball. We achieve this by configuring another section for all OneFootball users (having @onefootball.com as their email)

and wire this up with the Feature Gate

4. Gather feedback from the larger audience

Deploying the revised feature to production — available to all users

After gathering all the feedback there was, we incorporated it into the feature and deployed it for all the users out there

Takeaways

With this whole process, we learned some very important lessons.

1. Collaborative and Fun

The most important thing is that making it available only to the team, and requiring feedback to move forward required the team to collaborate and bring forward ideas about what they think about it.

2. Early Feedback and Quick Actions

Because we received early feedback, we were able to address the action items without feeling anxious about releasing something that could have been improved.

3. Releasing Better vs Releasing Faster

Even though there were feedback cycles, within and outside of the team, which took some time — we were able to release a better version of the feature.

4. Confidence in Deploying Directly to Production

We were deploying to production frequently because we were in full control of who sees this and who doesn’t — once we thought we’d improved to the best of our abilities, we launched the feature to all our users

5. We Learned a Good Way to Work

Yep, it felt good.