Which Git branching model should I select for my project?
It’s always great to start a new software project with a new tech team. At times we get the opportunity to join such projects from the inception. However, most of the times, we do not have that luxury, so we get to join projects that are half way done by someone else.
Few months back, in one of such projects, I had to take-up the challenge of setting up a new development process for the software team. By the time I joined this project, the team has already pushed the first release to the customer.
My project team was not a small single scrum team. The team had 25 software developers and also couple of QA engineers. The need of the management was to increase the team size in to 35- 40 developers parallelly working in 3 main product verticals. The team was following some of the scrum of scrum practices to collaborate with multiple teams.
The development team had various levels of skills. Some were just fresh from the universities while some other engineers had 10+ years of experiences in the industry. However, one good thing was that, most of these members have worked together previously and they had pretty much good team dynamics among themselves.
On top of that, customer was having serious market pressure to release the product as soon as possible.
This was a complex situation for any technical lead to get his or her hands in. Proper technical practices and governance are critical aspects of such contexts to perform as a high performing team. Therefore, my task was to establish required practices and governance, but I had to work super fast to get things in place before the next release.
I started looking at the source control in place.
For the source control, my team was already using Visual Studio Team System with Git. However all the development was carried in master branch which I had to change soon.
With the new development process to be introduced to the team, we agreed to achieve the following;
- Code merging to stable branch (in our case its QA) must be thoroughly reviewed
- Incrementally release to QA, Staging and Production environments
- Only completed User Stories should be merged to QA so that QA can test feature end to end
- Facilitate release types of QA , Staging, Production and hot-fix.
After considering all facts and challenges, we came up with the below branching model.
dev branch: This is a special branch in our context, where it will be only used as an integration branch if developers want to deploy and test on Service Fabric cluster before releasing to QA. (yes, this is Service Fabric application)
qa branch: This is the stable branch where all commits are integrated and release to QA. Code can be merged to QA branch only via Pull Requests as we had to control what comes to QA branch.
stage branch: Sole purpose of this branch is to release to Stage environment for Acceptance testing.
master branch: All production releases should be done from the master branch. Each release to be tagged.
Hot fix branch: Branch off from master branch. Upon Releasing hot-fix, changes are cherry picked to Staging and QA branches so that it’ll go with subsequent releases.
Feature branch: Branch off from QA branch. Once feature development is completed, Developer will raise a Pull Requests to QA branch to merge changes to QA. A pool of designated reviewers were assigned to review the code before merging to QA.
Code was incrementally merged from QA to Staging and then to Production branches to do Staging or production releases.
Learnings
As described earlier, the team was already practicing trunk-based development. Although the new approach addressed the set objectives, in retrospective we understood that it made things more complicated and slower.
- Some argued that Pull Request originated from Open Source world where anyone can contribute to code, which needed a thorough code review process before merge code to mainline. Do we need it here?
- This approach also contradicts with the core principle of Continuous Integration (CI) as it intentionally making integration slower by adding more gates.
- How about code reviews? Are they effective? there were pool of reviewers available, but they had no much idea of the features. Does it make sense?
Apparently, all these were valid concerns. I’ve managed to capture the continuous team learnings and eventually establish an improved development process.
Still I felt we could have done better with our approach. Following questions started to pop in my mind.
- Can it be simpler?
- Did we get the full use of Azure DevOps?
- Should we have stick to trunk-based development and find way to control changes to trunk using a review tool?
Problems with the above branching model
This development process greatly helped us to keep QA branch stable. However, I felt that the amount of time teams had spent to maintain feature branches, Pull Request reviews and conflict resolutions were too much of a cost to achieve the stability in modern day software development. Some of the major drawbacks I saw in the workflow were as follows;
Long lived branches
Though the workflow promotes on creating short-lived (single sprint or less) branches, you may still see some long-lived branches in the repo after a few sprints. Only the developer would know why the feature branch still there and when it’ll be merged to QA branch. Eventually it has been resulted in too many conflicts and wrong conflict resolutions when merging to QA branch.
Branch per environment
Keeping branch per environment is an Anti-pattern. Major issue with this approach is, though the releases are promoting through pipeline through QA, Staging and Production branches, we’ll always be testing different set of binaries since each CI pipeline rebuild artifacts upon merging code to respective branch. This gives false security of the deliverables when there are many ways that could go wrong. It could be a configuration issue on branch, or it could be conflicts due to additional commits directly done for hot fix releases which was not Cherry-Picked to QA branch. As a result, this approach has lowered confidence about the release artifacts instead of improving it.
Code Reviews
Apart from the fact that reviewers lacking the knowledge about the features coming for reviews, Pull Request itself only show the changes related to the relevant commits. Hence the reviewer doesn’t get the full picture unless he is already known about code. Most of the times code reviews been limited technical scope, whether feature delivered the business scope given in use case been rarely reviewed.
What is the alternative…?
Alternative approach would have been the Trunk Based Development (TBD) as we have done at the beginning with some tweaking. However, for us to continue with trunk-base development as long-term process still questionable and might have led to different avenues.
Let’s look at what factor decides when adapting either of workflows.
Git Flow
Main advantage of the Git flow is the strict control. Pull Request can be used to limit and control access to branches. Same advantage is also could become a disadvantage as it slows down the continuous integration of code that eventually result in slower feedback cycle. You’ll have to compromise one with another depending the on the nature of the project and the team. If the team consists of many junior developers, it’d be good to review code before merging to master at least until confidence about the code quality is satisfactory. However, if the team is well experienced and had worked together previously, you don’t need to slow them down with unnecessary gates, rather checking the issue if it arrives.
Trunk Based Development (TBD)
Trunk Based development is super simple. All developers commit to the same branch. Usually the master branch. Then deliver all the way to production using pipelines of deployment stages. DevOps tools facilitate required visibility of builds and artifacts. Main advantage of this approach is the quick integration and feedback cycle-time. If something goes wrong due to your commit, you’ll immediately known via CI Build failure. Also, it’ll completely avoids the merging horror.
Release pipeline would be set only on the mainline and deploy to different environments via deployment stages with controlling with release gates. Modern DevOps tool can give highly visualize view of deployment stages. Major different here and running different branch per environment is the release artifacts. Binaries are only built once and releases to different environment with different configurations. I’m sure we all can agree that we’ll have more confidence of releasing these artifacts to production than the branch-based artifacts.
Tough it looks simple approach, you’ll run in to below issues during the execution.
- How do we review code without committing code master?
- How do we avoid releasing incomplete feature development to production?
Code Reviews
Code review is tricky part in Trunk Base Development. Easiest approach would be to do the peer review prior to committing code to master. This discipline is effective if the team is matured and familiar with each other as discussed in above. Otherwise it may lead to long delays to get peer review even for most obvious code change. Also, there could be practical issues such as “Working from home” scenarios as most of companies are now allowing employee to work from home.
Other approach probably would be to use short lived feature branches to make the code review process easier. This will make integration slower but probably not slower as GIT Flow.
Feature Flags
Feature flag is just a switch in source code to simply decide feature is active or not. By default, all incomplete features are disabled. This allows feature to be disabled in production and enable in QA or development if needed for testing. Team can continue to commit to master though the feature is not fully completed.
Conclusion
Both Git Flow and Trunk Based Development has its own pros and cons. Selection between two carefully though of by taking many factors in to consideration including level of team skills, complexity of the projects, time to production etc.
One major factor that I’ve overseen in above experience was the business objectives. Business objectives of your customer and of your company (if you are a solution provider for another company). Because we introduce these flow and process to solve a business problem eventually. Hence, it does make sense to know what part of business objective would be addressed by adhering to these processes & practices.
Trunk Based Development could be tricky, but it should be the go-to approach if the team is well experienced, worked together for considerable amount of time and if the product itself doesn’t needs strict control. Also, you can make it more fun by adding few team felonies such as,
“if you commit code wait until build is green with acceptance tests passes”,
“do not commit to broken build, wait until build is green”
If you are in the other end of the spectrum where team is inexperienced and new, you should probably consider going for Git-flow with short lived feature branches. But make sure to delete feature branches when completing Pull Requests.
Use deployment stages instead of using environment specific branches to avoid rebuilding of release artifacts. So that the team can confidently release artifacts to production.
Next time when you are in a situation to propose a development process for a project, don’t just select practices or tools because of the hype or someone else is using it well. Make sure to invest time early on decision process and consider all these factors within the context of your team skills, complexities and business objectives.
