Avoiding Build Breakage — Essential Practices for Continuous Integration

Build breakages are expected as part of our day to day development tasks when the code is transforming and evolving rapidly. When a lot of people working on a project and code base is highly distributed, build breaks are inevitable. We need to plan for build errors and should have a proper approach to find and eliminate them as soon as possible. With our strategy, we should be able to block build errors propagation to other development and deployment environments and address them in place quickly as possible. When the build is passing, we should be able to guarantee that the code should be able to build on any environment that meets the required setup.

The basic idea of continuous integration is to integrate code changes more rapidly. Each integration is verified by an automated build with a test suite to identify possible build or functionality errors. Each developer is expected to integrate their changes frequently — the frequency may depend on what the development environment is. If you’re the only one contributing to a code base, then a deferred commit won’t be that crucial. However, if you’re in a team of several developers, and everyone thinking “oh well, I’ll wait with the commit a little” then you’ll often end up handling a lot of conflicts and losing a lot of time.

The general rule (for both scenarios) would be: Commit as often as possible. If you think “it’s not ready yet” (because it’ll break the build or simply isn’t done yet) then create a branch and commit to that branch but make sure you do commit.

Also, at the end of the each task, developers are expected to fetch the latest changes from the origin of the repository. These practices help teams to significantly minimize integration problems among the other benefits.

With proper continuous integration in place, developers are always starting their feature developments from a known stable code base which guarantee the proper functionality using an automated test suite. If CI build is broken due to a seldom factor, it is very important to fix it fast as possible to guarantee the reliability of the main branch.


How to avoid breaking builds

There are some golden rules to avoid breaking builds in an agile, dev-ops friendly environments.

Commit smallest changes possible

It is very important to commit your changes frequently. When you are done with a certain feature or functionality, commit it and push to the integration branch. To do so, you have to update your working copy to match the integration branch and resolve any conflicts locally. If the build is passing with the commit test suite, you are good to commit. Do not wait till the whole story gets completed. It is essential that developers check in their changes as soon as possible when they are working with a team to maintain conflict free healthy integrations environment.

Avoid end of the day bulk check-ins

Research revealed that many integration build issues are happening around 5 p.m. to 6 p.m. duration. This is because that many developers accumulate all the changes they did throughout the day and try to check in them at the end of the day. This may lead to a lot of conflicts and build breaks. Also fixing the issues will be a tedious task at the end of the day with considerable changeset which drives them to seek quick fixes.

Make sure the build is intact, before check-in

Always verify the build status before doing a check-in. If the build is failing you have to fix it locally before pushing changes to the source branch. More importantly, if integration build is failing, you have to wait until it is fixed to push your changes. Do not check-in over broken builds. It may complicate the fixing of integration branch and your changes may introduce bugs and undesired effects after the integration branch is fixed.

Run commit test suites locally before check-in changes

Define a commit test suite to align with your feature work. Always run that suite and verify the correct functionality before pushing your changes to upstream. Never intentionally pollute the integration branch with faulty code since that changes may highly likely to propagate to other developers before you are able to apply a fix.

Complete all the steps of the commit cycle

Committing your changes to upstream does not finish your development cycle. You have to wait until CI build job is completed to see how your changes are integrated with others changes. Also look for unit test results to identify any functional bugs. You need to attend immediately if issues are identified.

If you break it, you make it!

If integration build is broken due to your change-sets, you have to take full responsibility to fix it as soon as possible. Owner of the change-set is the best person to address the issue effectively. Do not wait till next day to fix the issue. Consider build breaks as highest priority items since they can spoil the other developer environments and also, will interrupt the team’s development workflow significantly. If fixing integration build take considerable time, revert changes and bring the integration branch to the previous status. After the fix is applied locally and tested properly, you can push your changes back. Take the full ownership of build breaks happening due to your code changes. you are the best person to attend them and also should be your number one priority.

Do not tweak build failures to make it work

When fixing build breaks, apply a proper fix. Do not tweak fixing build breaks for the moment by commenting out code segments and applying quick patches to overcome the failure. Take your time for a proper fix.

Enhance CI workflows to mitigate build breaks

Proper branching strategy

Proper branching strategy is a key thing to avoid unnecessary conflicts and build breaks. With proper branching strategy, developers can easily sync with the main integration branch and fix possible conflicts and issues locally before check-in the code. Git workflow is one of the popular branching strategy used for CI enabled git repositories.

Build notification system

Continuous integration build system should be configured with the proper notification system. If build is failing, notification system (eg: email alerts, hipchat notifications) should be able to notify team and responsible developers immediately.

PR build system with prior merging

One of the best places to avoid build breakages is pull requests. We can configure pull request build system with prior merging. In that case, pull request will merge virtually before the build and merged code will be input to the build job. We need to run commit test suite with the build job. So we will be able to identify build breaks and functional breaks before we pushing changes to main integration branch.

Image is copyrighted to Geek & Poke under a CC License