Are we Accelerate-ing yet?

Published in

Welcome Tech

10 min readMay 17, 2022

Some of you, dear readers, may already be familiar with the book Accelerate, published in 2018 by Nicole Forsgren, Jez Humble and Gene Kim, all well-known authors in our industry.

It expands on surveys carried out by the authors over several years while they investigated the code-delivery culture in roughly 2,000 companies around the world. Their goal was to find the traits and practices that may or may not imply performance gain compared to other companies.

Spoiler alert: They discovered that high-performing companies almost always use the same set of practices. After examining the results of their surveys and analyses, they grouped those “best practices” into the following sections:

Continuous delivery
Architecture
Product and process
Lean management and monitoring
Cultural

In this article, we will focus on the continuous delivery section and see how we are performing here at Welcome to the Jungle when it comes to those practices.

Please do keep in mind that as a back-end developer, I will be very back-end oriented, but our front-end teams of course also take these subjects seriously in their daily work. A bit of context before we start: Regarding back-end development, we have one Ruby on Rails monolith and a dozen services written in Elixir using the Phoenix framework. All new services introduced in our system tend to be written in Elixir now.

Version control

Let’s begin with the obvious one: using a version control tool.

Git is everywhere now and taught on almost every course in the world. GitHub has brought us social features for core git capabilities and useful tools, such as pull request review tooling, improving our developer experience on a day-to-day basis.

More precisely, as Accelerate highlights, there are four artifacts that are likely to be version controlled in high-performing companies:

The application code itself
The system configuration
The application configuration
The build and configuration automation scripts

The book underlines the fact that having system and application configuration version controlled is more often linked to high-delivery performance than having the code under version control.

So let’s get started and check how we behave regarding these artifacts!

Application code

I feel pretty confident that you won’t be surprised to find that all our application code is under version control using git on private GitHub repositories. We even have some public repositories for our open-source projects. Currently, we maintain two open-source libraries: Algoliax, since we are an Elixir client for the Algolia search engine; and Welcome UI, a design system, and its corresponding React components written in TypeScript.

Each of our services sits in its own git repository.

System configuration

Somewhat less of a common thing, we have our system configuration in the same repository. We deploy our applications to AWS and keep related application configuration files side by side with the application code.

Regarding our infrastructure configuration we have a dedicated git repository containing all the configuration required to set up our EC2 instances, RDS databases, and so on.

Besides AWS-related configuration, we also keep our sensitive data and environment variables directly in our git repositories. Don’t worry, we encrypt these files to keep them safe and secure using SOPS. Remember when I said we have different back-end tech stacks? Using SOPS helps us to use a common and agnostic way of sharing sensitive data across our apps, even for our front-end.

Application configuration

As we are using Elixir and Phoenix for pretty much all our back-end services, all our application configuration sits alongside the application code, mostly written in Elixir using its Config keyword-based configuration API.

We tend to use application-related environment variables in our application configuration in order to ease any future use of those variables in code, as shown in the following code snippet:

Build and configuration automation scripts

We use CircleCI to handle our build pipeline, from running our automated tests to building our application binary ready to be deployed. CircleCI uses YAML files to configure an application’s pipelines.

This configuration file is also located in the application repository and is therefore version controlled with git.

You can see a picture of our full build pipeline in the continuous integration section of this article.

Continuous integration

As Accelerate states, continous integration is the first step to continuous delivery. Almost all high-performing teams use short-lived branches, integrating them into the master branch on a daily basis. Whenever a change occurs in the code, the test suite is run and every failure is quickly fixed.

We rely on multiple automations in our GitHub pipeline to continuously integrate our code changes, as below:

We run our test suite (of course).
We run static code analysis to automatically find bad smells in code changes using credo.
We check our syntax and code-formatting rules with a linter using the mix format command.
We also have a security-focused static analysis tool called Sobelow running.
All branches are built in release mode, giving us a packaged application ready to be deployed in our preproduction and production environments.

Trunk-based development

During their research, the authors of the book found that the highest-performing teams tend not to use feature branches and follow trunk-based development, working directly on the trunk/master. More generally, well-performing teams use short-lived branches, keeping the number of branches that are active at the same time limited to 3, each branch being open for less than a day on average.

Talking of code-versioning practices, we do not follow a trunk-based development yet. We have steps in our development pipeline that mean we are able to test a specific set of commits involving multiple repositories (for example, the integration of our back-end and our front-end, two services communicating via an internal API or using a message broker).

Our current efforts regarding this entail lowering the scope of changes included in each branch that we maintain alongside the main one. Those efforts may involve the use of feature flags, complex migration plans with multiple deploys, or API versioning. All of these practices will help us to migrate to a trunk-based development someday if needed and to continuously reduce the time a branch stays open, thus reducing the total number of active branches we have open at the same time in one repository.

At the moment, we use a simplified git flow process regarding our branching strategy, with some minor differences that suit our internal processes better.

Test automation

The results of the survey regarding test automation revealed mainly two things:

High performers have a resilient test suite. Such a test suite improves developers’ confidence when it comes to deploying code changes into production. On the other hand, a resilient test suite failing to run flawlessly is an indicator of a real defect.
Organizations that are high performing have their software engineers writing tests. They are also able to run the entire test suite on their own workstations, enabling them to quickly fix broken tests.

At Welcome to the Jungle we utilize the usual suspects when it comes to our testing strategy:

Lots of unit tests.
A good share of integration tests for API endpoints.
End-to-end tests using Cypress to ensure that our front-ends and back-ends speak the same language and that the plumbing between our back-end services does not leak!

Our unit tests and integration tests are run by our software engineers, while our end-to-end test suite is shared between them and our QA engineers.

Regarding test coverage, we are currently introducing a step in our CI that won’t accept any commit that lowers the percentage of code covered by our tests. At the moment, the two back-end services owned by the squad I am part of have a test coverage of ~75%, which we are constantly trying to improve.

More than asserting that our code works, our test suite helps us remain confident that our code is still working as expected while we introduce code refactorings.

Test data management

Accelerate highlights the fact that high performers have adequate test data, enabling them to run meaningful test suites. Additionally, they can easily access test data on demand and are never limited on the number of tests they can write because of a lack of test data.

In our teams, test data and database seeding is an ongoing point of enhancement. We are considering a few methods so that we can use a working set of anonymized data from our production database in our seeds. This would help us work with meaningful data while developing new features and also improve the quality of our end-to-end test suite.

Another good side effect of having great seeds is the improvement it would make to our onboarding process for newcomers. Being able to run the system locally and instantly see meaningful data in the front-end is an extremely valuable thing to have on day one at your new company.

Shift left on security

The authors of Accelerate found that high-performing teams incorporated security during all their development processes, from design to delivery. A commonly held belief is that including those concerns during the software-delivery life cycle slows down the entire process, but that was not the case for those teams.

We of course take security very seriously at Welcome to the Jungle and have recently improved and hardened our practices regarding it. We have sensitive and personal data on our databases that we have to keep protected from any sort of breach.

Here are some examples of the improvements we have recently introduced:

We conduct internal security audits to find breaches that may have made their way to our production servers and fix them quickly, before any data can be exploited.
We keep an eye on security announcements and share them in a dedicated Slack channel.
Some software engineers train themselves on the subject to be proactive and ready to correct possible security holes before they hit main branches.
We hold company-wide training sessions to make all employees aware of security issues and needs.

Security is a subject shared and owned by multiple coworkers across many teams, thus limiting the knowledge silo and spreading best practices across the company.

Deployment automation

The authors note that the main advantage of applying all the previously mentioned best practices is that this enables the automation of the deployment process. Teams do not delay releasing code changes to the production servers, even during business hours.

Our fabulous SRE and platform teams have built an internal CLI to handle all sorts of automations, including the deployment process. We are one shell command away from deploying our apps in all our environments (production, preproduction, qa) and it almost always looks like this:

$ wttj deploy preprod — current-branch

We use the blue/green deployment pattern, starting a new machine with the new code running alongside the previous one, and waiting for the former to be fully operational and network-wise before killing the latter.

Recently, we even added sprinkles to this CLI tool and it now automatically sends a Slack message on a dedicated channel to inform the team of every deployment, giving the direct link to the CodeDeploy page related to it:

For the time being we are not automatically deploying to our production environment nor to our preproduction environments, mainly because our infrastructure is not ready yet.

Currently we have five preproduction environments for each service that we operate, meaning that we have to remember that one preproduction environment is already being used to validate the work of another team.

Also, our SRE team is installing Kubernetes (or K8s), which will allow us to significantly improve our deployment speed and reduce our infrastructure cost. Another good side effect of using K8s is the ability it offers to deploy feature branches in their own isolated environment, pushing forward the testability of our information system when it comes to the integration of multiple services. But this extra step requires a lot of work and will be prioritized later in our roadmaps.

Our newest services are already K8s-ready, acting as proof-of-concept and tracing the path for our older ones.

Conclusion

As you may have already noted, we are not yet completely on a par with high performers regarding our continuous delivery processes. But of course we are always striving to improve our way of working and are constantly enhancing our migration process to reach our continuous delivery goal while taking into account code-quality metrics and best practices regarding our version-control processes.

Below is a visual representation of our current performance in continuous delivery capabilities: