Continuous Integration: Is it possible in Branch-Based Development?
Continuous Integration — the term seems self-explanatory. Integration that happens all the time. In the software development context, it’s the combining of application source code changes into the main or master version of the code base. This indicates that some form of code versioning must be used, and the main version of the source code is often referred to as the mainline, trunk, or the master branch. Continuous integration (CI) is focused on ensuring code changes are frequently integrated with the mainline and ready for deployment.
Achieving and maintaining deployment readiness is what makes CI feel more like an art than a science. It definitely calls for strategic thinking as oppose to being reactive. Achieving it is not about harnessing expensive and complex tools, as Martin Fowler points out, “The essence of it lies in the simple practice of everyone on the team integrating frequently, usually daily, against a controlled source code repository.”
The major components of the traditional CI model are:
- Maintain a single source repository of code.
- Automate the source code builds. “Traditionally a build means compiling, linking, and all the additional stuff required to get a program to execute. A program may run, but that doesn’t mean it does the right thing. Modern statically typed languages can catch many bugs, but far more slip through that net.”
- Run automated test. Test must be extensive and give significant confidence to developers about the state of the code base. Low level automated test, like unit test, are usually triggered by a pre-push git hook (or something similar). This means that those test run on the developer’s machine and must pass before code is pushed to the mainline of the remote code repository.
- All developers commit directly to the mainline (usually called the master or develop branch). This is known as trunk-based development.
- Everyone commits to the mainline every day. In other words, you integrate your code changes into the mainline daily.
- Every code change to the mainline should be built and tested.
- Fast feedback. Keep the build and test feedback fast, giving developers quick status updates on the source code health.
- IMMEDIATELY FIX BROKEN BUILDS AND ADDRESS ANY FAILING TEST. This is vital to achieving the underlying goal of having a healthy code base — ready to be deployed at a moment’s notice.
- Test in a production-like environment, which could involve automating deployment.
- Make it easy for anyone to get the latest application executable.
- Everyone should be able to see the state of the mainline. So, if the latest build has failed on the mainline, developers will not update their work by merging with the mainline. Communicating the state of the mainline is a critical part of CI.
Key things to note in what I refer to as traditional CI:
- There are no feature branches. As mentioned above in point #4, trunk-based development is used. This is reflected by working directly out of the mainline. The project is initially cloned and the developer starts working on the mainline on their local machine.
- There are no code reviews in a pull request because there is no pull request; however, pair-programming makes up for this absence, and is arguably more effective.
- Frequent integration is key, as mentioned in point #5 above. The code that a developer is working on is committed often (at least every day, if not several times per day).
My Observations Over the Years
The software developers and development teams I come across usually use branch-based development (BBD); this refers to using feature branches, as opposed to trunk-based development (TBD). I think this is mainly due to how companies choose to approach reviewing source code in the absence of pair-programming. Without pair-programming, code reviews are critical to ensuring high and consistent code quality from the standpoint of design patterns and overall system architecture. Developers rely on pull request to coordinate code reviews, often giving and receiving written feedback on their code changes. So, changes are not merged directly to the mainline, which is the central source of truth, but integrated with it via pull requests after the code changes are approved. Usually, some level of build and test automation is run as part of the approval process as well.
Basically, with BBD, using git-flow or a slight variant is very common. In summary:
1. Developers use more than one mainline. There is not only a master branch, but also a develop branch. Originally the develop branch is created from the master branch.
2. Feature branches are used to create specific application features and these branches are created from the develop branch.
3. When a feature is complete the developer working on that feature creates a pull request to have that feature reviewed and then merged into the develop branch.
4. When teams want to release the application to the end-users, a release branch is created from develop. After the release, the release branch is merged into the master and develop branch.
5. If a bug or an issue is found in the production application, a hotfix branch is created from the master branch.
6. When developers are finished with the hotfix, essentially when a new version to the application is released from the hotfix branch, the hotfix is merged with the master and develop branch, as well as any existing release branch.
The commits on the master branch are usually tagged. The tag corresponds to the version number of the application that has been released. It’s by tags that developers can easily find a previous version of the code base and check that version out if required.
In addition to the major points above, some development teams work with epic branches. Epic branches are long-lived, similar to the develop branch. The epic branch is a space for fashioning a major feature set of the application, and teams often prefer to keep these features separate from the main source code until the feature set is completely functional. This is a major departure from continuous integration. What is even more unfortunate is that some teams using this approach still believe they are achieving continuous integration. Remember true CI leads to deployment readiness.
What to Aim for When Doing Branch-Based Development
No matter your branching strategy you should still be able to practice continuous integration. The key is to understand the goal of continuous integration and why the steps outlined in the traditional CI model are important.
“Continuous integration is a practice, not a tool. It requires a degree of commitment and discipline from your development team. You need everyone to check in small incremental changes frequently to [the] mainline and agree that the highest priority task on the project is to fix any change that breaks the application. If people don’t adopt the discipline necessary for it to work, your attempts at continuous integration will not lead to the improvement in quality that you hope for.” [p.57, Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation by Jez Humble & David Farley]
Software development teams with purpose can still use the git-flow approach and achieve a high level of integration if they truly understand what continuous integration is, and what makes the traditional approach work. One of the key ingredients to succeed, “You need purpose, a great idea around which you can organize yourself and your passion.” [p.315, Total I Ching: Myths for Change, by Stephen Karcher] That looks like this: having an application that is always deploy ready from the mainline. Teams need to be open to working by the principles of CI and must have the support and resources they need. It calls for increase involvement from developers; time and energy; think augmenting the code base — commonly known as refactoring! “Augmenting continually enriches things without setting up structure.” [p.314, Total I Ching: Myths for Change, by Stephen Karcher]
Developers should be humble and modest as they help each other and unconsciously harmonize as a team; give, encourage, assist. The team just needs to keep things simple and remain connected to the fundamentals of CI.
So, even with BBD, nothing stops developers from having test automation that can be triggered with a pre-push git hook, as well as having the build and test automation run when pull request are created. And nothing stops developers from only submitting small code changes in their pull requests and arranging for every commit on the mainline to be built and tested with test automation, thus making the project self-testing. You can make small changes and commit often to the mainline, even if it is through a pull request. When build and test automation on the mainline fail, it should be fixed immediately. If the problem cannot be resolved in 10 or less minutes, the code changes that caused the failure should be reverted. This allows other developer to continue to work with a stable application.
But let’s not drop the ball by forgetting about the epic branch. Teams that use epic branches would essentially be using another long-living branch, similar to other mainline branches like develop. For the epic branch, they would need a build and test automation strategy identical to the one used on the mainline. They do run the risk of having their continuous integration fall apart. The epic branch is not being integrated to the mainline, which is often the develop branch. So true continuous integration no longer exists. Also, by the time the epic branch is ready for integration the code changes are significant and extensive. It’s a recipe for merge conflicts and broken builds. The life of the epic branch must be thoughtfully planned and choreographed as it relates to the main code base. So why have an epic branch?
Feature Flagging and Branch-by-Abstraction
Most developers are not comfortable merging incomplete features into the mainline. And to the complete a feature can take 3 to 7 days. Out goes the commit often mantra — integrating your code frequently. This, however, shouldn’t be the case. Let it be known that code bases carry hidden and incomplete features as a norm. Feature toggles and branch-by-abstraction are brilliant ways to introduce hidden features and large-scale code changes to your application via the mainline. But even before employing these techniques, break down the work required to create the feature into small batches of code changes. This simplifies committing the code changes to the mainline and ensuring continuous integration. Your changes should not disrupt the working application, and releases should still be possible if desired. When it comes to the large-scale changes that tempt us to use an epic branch, use branch-by-abstraction to keep your continuous integration intact.
Perfect Example of CI at Work
And finally I leave you with the perfect display of CI in action:
“Your authors once worked on what we believe to have been, at the time, the largest agile project in the world. This was a geographically distributed project working on a shared codebase. The team as a whole was, at various points in the life of the project, working simultaneously in San Francisco and Chicago in the USA, in London, UK, and in Bangalore, India. During any given 24-hour period there were only about 3 hours when someone, somewhere in the world, was not working on the code. For the rest of the tome, there was a constant stream of changes committed to the version control system and a constant stream of new builds being triggered.
If the team in India broke the build and went home, the London team could have their day’s work dramatically affected. Similarly, if the London team went home on a broken build, their colleagues in the USA wold be swearing under their breath for the next eight hours.
Rigorous build discipline was essential, to the extent that we had a dedicated build master who not only maintained the build but also sometimes policed it, ensuring that whoever broke the build was working to fix it. If not, the build engineer would revert their check-in.” [p.69 Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation by Jez Humble & David Farley]
