Git Feature Flow

An improved Git branching model based on production ready feature branching

This article is targeted at software and web developers and technical managers and outlines a different code version control approach to handling deployments in fast moving, agile environments

The problem

As developers, we have all been in the painful situation where a scheduled release can’t go out on time because one or more of the included features hasn’t completed testing. The problem is all the included features have already been merged into the release branch for testing which means now the whole release is not ready. The reason for the delay could be last minute bugs or simply that the time to develop the release was underestimated, there could have been technical roadblocks or it could even be that key stakeholders haven’t provided enough information to complete a feature. But the result is the same, we can’t hit the deadline.

With our tail between our legs, we go to the client or stakeholders to let them know the bad news and they inevitably ask for everything else to be released without the features in question. It can’t be that hard right? Most of the time they have a very valid reason for asking for this as they have a massive marketing campaign going out or they promised management that it would be up today. Painfully, the answer is it is usually quite hard to remove features from a release as the code within that release branch has already been merged for testing. Cue the red bulls and a late night.

The pain of trying to remove features from a release branch

Using a release branch based methodology in Git, we have a couple of options when trying to remove a feature from a release:

1. Revert the code commits related to the features in question; or
2. Start a new release branch and merge only the features that are ready into that branch. This requires that you were doing feature branching in the first place.

From our experience, it is best to avoid reverting commits at all costs as it creates problems down the track. The reason for this is a revert in Git doesn’t change the commit history, it actually creates a new commit taking the opposite actions (e.g. if a semicolon was added to line 34, remove that semicolon now in a new commit). What this means is even if you are using feature branching, which is a great method of encapsulating code related to individual features, you run the risk of permanently reverting code changes if you revert a commit, even if you later merge in that feature branch again, once it is ready. The reason for this is the commit that the revert command created is the latest action that happened to the lines in question (e.g. the history says the semicolon was added then removed later). So as a general rule, never do reverts in Git. Another option here is to use rebasing to actually remove the commit from the history but this falls over horribly if you’ve since merged the branch into another branch or created new branches from this branch. This is especially problematic in Git as it’s a distributed repository setup and it might be that someone else has done this and you might not even know. Perform rebasing at your own risk.

Starting a new release branch if your current release branch is not releasable is completely feasible and what we often found ourselves doing, but the bigger your releases, the more difficult and error prone the process is and can often involve cherry picking commits which results in a loss of commit history. After doing this a few too many times, we thought “there has to be a better way!”. The reason we kept getting ourselves into this situation was most software development lifecycles, including Git Flow, are based around the concept of incremental releases which includes a set of agreed upon features. A lot of planning usually goes into what is to be included within a release to ensure the release schedules can be met but the inevitable happens far too often and delayed features result in delayed releases. We even toyed with the idea of setting up test environments per feature branch to avoid this situation, but we decided against it as it was not only an expensive proposition that would also be confusing for clients when performing acceptance testing, but it also doesn’t provide any integration testing.

The solution

There had to be a better way! Surely there was a way that would allow us to release whatever features had passed testing on the release day without having to worry about removing incomplete features

To give you some background, I am the Technical Director at the Melbourne office of Deepend, a leading Australian digital agency. We work on many projects in parallel with a large number of clients. We migrated to Git a few years ago and immediately started doing feature branching as it was an amazing way to stop developers stepping on each other’s toes without the massive pain of merging that existed in SVN and previous code versioning systems. We were all in awe of the advanced merging options that came with the change to record actions within files rather than just flagging files as changed. But even with feature branching, we still had the problem of releases not going out on time and upsetting clients. We started using feature branching and quickly progressed to Git Flow but soon figured out that whilst it was a great way to encapsulate features during development, it didn’t allow us to release features independently as each feature branch comes off the development branch and often includes unreleased code from the development branch. So we came up with a solution that involves creating feature branching from production that is similar to creating a hotfix in Git Flow and we have been evolving that solution ever since. Recently, we thought it was ready for the world, so we gave it the code name of Git Feature Flow, turned the scribbled diagrams into a sexy poster (download as A1, A3 or A4) and I’m writing this article in the hope helping other people facing the same problems.

Whilst Deepend is not a completely AGILE agency, we have taken a lot of inspiration from this approach to running projects and as such we run our builds in iterations (aka, sprints) so we are delivering early and often and testing as we go. Each sprint is effectively a new release. Given reliability is a key tenement of a good client relationship, we deliver our releases on time. If we have to delay a feature or two to hit a deadline, and the client is happy to do so, we do. Git Feature Flow allows us to do that. In fact, GFF is flexible enough that it can be used for Continuous Integration as well as Continuous Deployment setups. We can’t take all the credit though as GFF stands on the backs of giants as it is a variant on Feature Branching that takes a lot of inspiration from the Git Flow methodology.

How it works

We’ve been using this coming on two years and it does work!

Git Feature Flow (Download Poster as A1, A3 or A4)

At Deepend, we call the dev branch Master, and have a separate branches for Staging and Production. These names are arbitrary and can be switched if your organisation labels things differently

A branch for each environment

Development, staging, production, etc.

We run a Continuous Integration setup via an automated build server and have a code branch matching each environment. This is a prerequisite of GFF and the code on each of these branches MUST match the code that exists on the servers themselves — they should never, ever be out of sync. All code changes go through Git and we do automated builds and automated testing from there. No manual deployments and no human error when deploying. So if a developer wants to release a feature to the staging environment, they merge into the staging branch and do a push. At this point the build server picks up that there is new code to be deployed on that staging branch and builds, tests and deploys the code to the staging environment. In fact, our developers never actually upload anything to servers directly which is great as it reduces the risk of a tested feature breaking when deployed to production because of an undocumented deployment step. The only reason a code branch and the matching environment should ever be out of sync is if the deployment is currently happening or it failed, at which point the developer who broke it is prompted them to fix it immediately. We have a funky dashboard in the office to let us know what’s going on:

Deepend Build Dashboard

All feature branches must be production ready!

The other key aspect of GFF is all feature branches MUST come off the production branch. A lot of companies call this the master branch (which is what we call the development branch at Deepend) but it is critical to GFF that this branch be the exact same code as has been released to the production environment. The reason for this is simple, we want to be able to release a feature at any time to any environment without dependencies on other features in development. The production branch will only ever contain production ready code that has already been tested and released so feature branches you start from this branch will not include any in development code. This is a key difference to Git Flow and it means any feature can be released as soon as it’s ready. As such, you no longer need to plan what is to go into a release as it becomes whatever features have passed testing in the allocated timeframe. At Deepend we still plan our build sprints and do release days as it’s a good way to manage projects but this same methodology could easily be used for Continuous Deployment setups.

Keep your branches up to date

It’s also good practice to keep your feature branches up to date with the latest production code to reduce the risk of merge conflicts on release days. It also keeps the branches production ready! At Deepend we’ve are currently working on a Git Hook script that automatically attempts to merge any changes pushed to the production remote branch into all other branches. If a merge fails, a notification is sent to the last developer that worked on that feature branch so they can resolve the merge issue. We find this approach works well as developers deal with production merge conflicts while the code they wrote is still fresh in their minds. It also means that the only merge conflicts that will happen on a release day will be conflicts between features in that release, as opposed to any conflicts relating to all the changes released to production since the feature branches in your release began their lives.

Release features to each environment independently

Using GFF, you need to forget everything you’ve learnt about software development lifecycles as you do not push your features to your development/release branch, then promote the whole release to staging, then promote staging to production. Instead you push the features into each environment individually, which removes their dependency on each other. For example, you could push four features to the development branch but only three pass internal testing, so you only push those three to staging, and so on. I’ve had a few developers argue that this is the downside of GFF when they are first learning it as it requires more merging but the benefits far outweigh the cost. But it is important to keep in mind as it can mean that merge conflicts might need to be resolved multiple times when merging into each environment’s branch. Given this, we actually get the individual developers that worked on features to do the merging when required. So if a production release includes features from three developers, we get the three developers to merge in their features to the production branch on the release day. That way, if a merge conflict is encountered it will be related to code that the developer doing the merge will have written and they will have previously resolved it so they will know what to do. We’d love to automate this step to replay the same merge conflict resolutions when going to the next environment but we haven’t figured that out… yet.

Only commit code to individual feature branches

All code relating to a feature should only ever be committed to that feature branch. Don’t mix your code within one feature branch. This is a basic feature branching rule to keep the code on each branch separate from each other and your main branches so they can be managed separately. It’s especially important for GFF as we also release features independently and we don’t want to inadvertently release other code when merging a feature branch in.

Never merge your development or staging branches into production

Never, ever merge your development or your staging branches into your production branch. In fact never merge your development branch into staging either. I know this is tempting as it’s what we are all used to with traditional release based software development lifecycles but using GFF, this is very likely to push untested/unreleased code to your staging or production branches. Remember that you should be doing automated builds off the branches.

Never merge your development or staging branches into a feature branch

For the same reasons listed above, never merge your development or staging branches into a feature branch as they will contain unreleased code that cannot be easily separated out from the feature code. This is the biggest error any developer new to this methodology will make and it’s a pain to resolve.

When restructuring your project or database, do this on all branches

This one is important, if you have to do any code refactoring that will move files around within your project or any database restructuring to do, you should actually do this directly on the production branch, without changing any functionality, and then merge into all the other branches. This goes against the rule about all changes going on a feature branch but it is a requirement as you want all your feature branches to be production ready, so if one of your features requires a change to the project or database structure, then make that change to all branches so the change doesn’t cause a million merge conflicts when after you merge the restructuring into your Staging branch (Git doesn’t handle files moving around very well when feature branching). This is probably the trickiest one to get your head around but it’s super important. The challenge here is you also need to make sure the restructuring will not break existing in development features or your production environment. Please note this same rule should also apply for any code first ORM changes you make that change the table structure of your database — this should be immediately implemented across all your branches to avoid runtime errors when switching back to branches that do not expect the database structure to have changed.

It is very important to still use a feature branch for the development work relating the feature that prompted the restructure. The key here is you want to ensure that all the features share the same underlying file and database structure to avoid merge conflicts and runtime errors BUT you still want to keep the feature branch separate from the other features so you can release it to the target environments when it’s ready

Enjoy happier clients

We’ve been rolling with this release methodology for nearly two years now and the result is happier clients as we are able to be more flexible when it comes to what is included in a release and as a result, be more consistent on our delivery dates. It requires buy in from all of your developers though because if someone breaks one of the cardinal rules, it can be painful pulling features apart. If everyone is on board however, it works really well and gives a lot of flexibility. I’d love to hear your feedback!