Streamline: A super-efficient branching and CI strategy

Do you want to Just Get On With It, but keep up the highest code standards? Read on…

Photo by Tim Gouw on Unsplash

What is Streamline?

Streamline is a combined approach for branching, continuous integration, and (optionally) continuous deployment. It starts small to get development teams up and running as fast as possible, and can grow when the project grows.

The approach has four main goals:

• Enable a blazing-fast development cycle
• Enable, but not enforce, a blazing-fast release cycle
• Increase developer confidence in the quality of code
• Reduce complexity of branches, environments and processes

In a rush? The 1000-foot summary

1. Limit branches to feature and trunk (equivalent to master ). bugfix is optional.
2. Use pull requests to trigger a CI pipeline that runs all functional tests and creates a Docker image. These functional tests are never repeated.
3. A fast-forward merge into trunk marks the Docker image as a new ‘version’ of your app.
4. The new versioned Docker image can be deployed anywhere, from ‘demo’ or ‘penetration test’ environments right through to production.

Here’s an example of how Streamline can be implemented, including Kubernetes-based deployments:

An example way to implement Streamline that has been found to work well in practice.

Not in a rush and want the full detail?

Let’s begin…

Mate, we all use GitFlow.

Indeed. And we all hate it sometimes, too.

How many times have you been on a project where your Pre-Prod and Staging environments have been so far behind your latest code sitting in the develop branch that you end up running demos from your Dev environment, or even on localhost?

Here are a few examples of GitFlow-related “…said nobody ever”s:

“I really love moving my code around between develop , feature , bugfix , release , hotfix , master …”

“Oh, you merged into develop ? Me too! I think our code will work together perfectly and not require any further testing.”

“There’s a critical bug in Production? No problem. I am super-familiar with the hotfix pipeline and can guarantee you, Mr. Stakeholder, that it will work first time.”

Sometimes GitFlow is necessary. But perhaps there’s something better for the many, many times that it just felt like overkill and overhead?

Ned Stark says it best…

Mate, how about Trunk-based development?

Trunk-based development (https://trunkbaseddevelopment.com) is simpler than GitFlow, agreed! In fact, it provided the inspiration for Streamline. However, it lacks a robust strategy for continuous integration, and can be simplified even further with a few tweaks.

Streamline is an attempt to have the best of both worlds: simplicity and production-level robustness.

Come on then, how does it work?

The core concepts of Streamline are in the branching and continuous integration (CI) strategies. Continuous deployment (CD) will be covered in a later section.

Branching

To get started, you only need two types of branch:

• One that represents the ‘single source of truth’, e.g. master or trunk
• One that represents ‘new code’, e.g. feature

For now, we will call the first branch trunk .

The branch type featurecan be split into multiple branch naming conventions if desired, e.g. bugfix as well as feature , but ultimately this name will be the only difference between them.

Here’s the basic workflow:

1. A developer branches off trunk to feature/excellent-thing
2. The developer makes their changes on feature/excellent-thing
3. Once complete, the developer raises a pull request back into trunk
4. Special and exciting CI happens (more detail later)
5. The pull request is fast-forward merged (more detail later)

…That’s it! As you can probably guess, we will be relying on the CI to keep code quality high.

Continuous Integration

The CI process must have two key components:

1. An automated pipeline that is triggered when a pull request is raised or updated
2. Merging the pull request into trunk is prevented until the following conditions are met:

• The automated pipeline has passed
• The feature branch is up-to-date with trunk

Both of these conditions are critical to the success of Streamline as an approach. Let’s explore each of them in more detail…

First merge condition: The CI pipeline has run and passed

The CI pipeline should:

• Run all automated functional tests
• Create a Docker image that contains the tested application
• Upload the Docker image to an image registry
• Optionally deploy the Docker image to a test environment

An example CI pipeline…. Blue = automated tests. Orange = image creation.

It’s important that all of your automated functional tests run as part of this pipeline. Include everything from linting to unit tests, coverage checks, component tests and UI/browser tests where relevant, but don’t worry about non-functional tests (e.g. performance tests) at this stage.

You won’t be running any functional tests later, so put them all in the CI pipeline.

The Docker image created at this stage is also key to the success of Streamline. The images created at this early stage must be ‘production-ready’, because they represent a releasable version of the application from the point of view of the development team and won’t be changed or rebuilt later.

Note that the CI pipeline must be re-run if new commits are pushed to a branch with an existing open pull request. In this event, previously created and pushed Docker images can be removed from the image repository because they are no longer required.

Second merge condition: feature must be up-to-date with trunk

This can be achieved in a number of ways, depending on your repository host.

For example, GitHub has an option called “Require branches to be up to date before merging”:

Others, such as BitBucket, describe the same functionality in git terms: when we git merge , there is an option called “fast-forward only”, or git merge --ff-only .

Using this merge strategy ensures that a merge from feature into trunk is only possible if the feature branch contains all of the historical commits from trunk . Therefore, trunk can be ‘fast-forwarded’ or ‘updated’ to be identical to your feature branch rather than creating a merge commit or squashing commits.

A subtle but important difference

By keeping the code identical upon merge in this way, we can guarantee that this version of our code does not ever need to be re-tested, and therefore this Docker image does not ever need to be rebuilt. That’s BIG news for us.

But how about if multiple PRs are raised, I hear you cry?

1. Bob raises a PR with one feature. The pipeline is triggered and passes.
2. Lisa raises a PR with another feature. Another pipeline is triggered and passes.
3. Lisa merges her PR into trunk first.
4. Bob’s branch can no longer be fast-forward merged into trunk .
5. Bob merges Lisa’s commits into his branch with git merge trunk .
6. Bob pushes his updated branch, which triggers a re-run of the pipeline.

This ensures that Bob’s code works with Lisa’s code, before Bob can merge.

Branching and CI recap

1. trunk >>> feature/excellent-thing
2. feature/excellent-thing >>> Raise PR
3. CI pipeline triggered. Build and automated functional tests run, and a Docker image created and pushed to an image repository.
4. feature/excellent-thing is fast-forward merged into trunk

At this stage:

• We have a tested Docker image sat in an image repository, ready to be deployed anywhere we like.
• trunk has essentially become a ‘tracking branch’ for our latest ‘releasable’ code
• The development team are confident in the quality of both trunk and the growing list of production-ready Docker images.

Continuous deployment (CD)

So you’ve got some great Docker images — now let’s put them to use.

The following steps are suggestions only, and not critical to the success of Streamline. However, we’ve found that Streamline lends itself very nicely to this particular CD strategy!

Let’s rejoin the story at the point of performing your git merge --ff-only into the trunk branch.

trunk has been updated, and this essentially signals a ‘new version’ in the eyes of the development team. What’s different in the case of Streamline is that we already have a tested Docker image that represents this new ‘version’, so all we need to do is (optionally) tag the existing Docker image with a version number, say 1.1.0, and deploy the image into an environment, which we call the ‘Trunk’ environment because it should always represent the latest code in the trunk branch.

This ‘tag and deploy’ can happen as part of the trunk CD pipeline:

There are three points to note here:

• A new version number can be calculated based on the most recent git tag or most recent versioned Docker image, e.g. 1.0.0 becomes 1.1.0 . You can take this further by assuming a Semver link to the branch types i.e. feature is a ‘minor’ bump to 1.1.0, whereas a bugfix is a ‘patch’ bump to 1.0.1 .
• It is possible to apply a new Docker tag to an existing Docker image. This allows the new information (a version number) to be applied to the image without changing the image itself.
• If a pull request is closed without merging into trunk (it might have been deemed unnecessary, for example), then there will be a leftover, un-versioned Docker image that will never be used. The CD pipeline can optionally remove these un-versioned images if necessary.

Scaling up and production

Small teams and projects in their early stages aren’t likely to need anything beyond pull request deployments and trunk deployments. However, as projects grow and progress, additional requirements usually start to appear.

For example:

• A separate testing team are brought on to cover non-functional requirements such as performance testing or penetration testing.
• The client wants to have a stable demo environment that doesn’t change or get updated until they want it to.
• Your app is an API that a second development team are building a front-end against.

For each of these scenarios, an isolated environment is required with a controlled version of the application installed.

Thanks to the nature of Docker images, we can meet this kind of requirement very easily: create a new environment, e.g. a new Kubernetes namespace called ‘demo’, then create a CD pipeline that deploys a specified version of your application into that environment.

This approach can scale to as many environments as are required, allowing any version of the application to be deployed or rolled back. Additional business process can be put in place where necessary to control or restrict these deployments.

The same approach applies when you make the move into Production. Blue/green deployments via a load balancer in a separate cluster is recommended, but the core principle of choosing and deploying an existing versioned Docker image of your application remains the same.

Hotfixes

Picture the scene: you’ve got a kick-ass dev team and you’re pumping out features into trunk every day. You’re up to version 1.8.3. However, the client is a bit slower than you’d like to approve releases, and Production is way behind on version 1.2.6 .

Suddenly, a critical bug pops up in Production, and all hell breaks loose. The client wants you to fix the bug without updating to 1.8.3 , because they’ve not approved those features yet. How can Streamline adapt to solve this issue?

The answer is to git checkout the tagged version number in your repository.

Git tagging should have been happening automatically as part of your CI pipeline. If it hasn’t, you should be able to trace back the Git SHA to the correct commit.

In this example, we would use:

git checkout 1.2.6

At this point, we are in a detached HEAD state. We’re looking at a particular point in time of our code, but we can’t make any commits because we’re not actually on a branch. This is easily resolved by creating a new branch:

git checkout -b hotfix/everything-is-on-fire

On this branch, make your changes to fix the issue, then git push .

Hotfixes are slightly different to feature/bugfix . The usual CI pipeline of ‘build, functional test, create a Docker image’ still happens in the usual way, but happens on git push (every time the hotfix branch changes), before a pull request is opened.

This is because we don’t want to merge our hotfix into trunk yet.

So, what do we call the Docker image?

In our example of hotfixing 1.2.6 , we call the Docker image:

 1.2.6-hotfix-<commitSHA>

This naming convention, with the inclusion of the git commit SHA, has the following benefits:

• Ensures a unique name if you git push more than once
• Allows multiple developers to try to fix the issue independently
• Does not interfere with the strict versioning happening on trunk
• Traceable back to the commit later

This hotfix Docker image can now be pushed to any environment including Test, Demo, Production etc. in the same way as any other feature or bugfix.

Let’s assume that after a few attempts, the client is happy with 1.2.6-hotfix-ff54s1 . They have run penetration testing against it and they think that it resolves the issue. The image is deployed into Production.

Your branch hotfix/everything-is-on-fire can now get merged into trunk:

1. git merge trunk (brings your hotfix branch up to date with the latest features in 1.8.3 and applies the fix on top)
2. Open a pull request into trunk

From this point onwards, the CI process is exactly the same as for bugfix — a new versioned image is created (1.8.4) and balance in the universe is restored.

In the unhappy scenario that the hotfix doesn’t work, the hotfix steps above can always be repeated by checking out the commit or tag, in this case 1.2.6-hotfix-ff54s1 , then creating another branch, hotfix/still-on-fire , and working from there.

Reasons NOT to use Streamline

Streamline isn’t perfect — there are times when it won’t be the best fit for your project. Here are a few examples:

• The dev team isn’t able to write fully automated tests that are high quality and provide high coverage. Streamline relies heavily upon QA to ensure that trunk is in the right state.
• The project uses an enormous monorepo for multiple parts of a wider programme, rather than smaller, microservice-sized repos. The requirement to always be up-to-date with trunk could slow down these kinds of projects rather than speed them up.
• The dev team or project is too large, slow, or risk-averse to effectively educate and instil the Streamline ways of working. It might be that GitFlow or another workflow is so embedded that the risk or cost of changing approach is too high.
• The tools that you are using don’t support the level of customisation required to implement Streamline, such as custom CI pipeline triggers, remote image registries, etc.

“I can see a flaw in Streamline”

Please let us know in the comments — we’re aware that this is early days for the approach, and there are likely to be issues that we haven’t come across yet.

Happy devving!