Continuous Integration

If it hurts, do it more often. — Martin Fowler

I remember years back being fresh out of coding boot camp and sitting across from a senior software developer in an interview. It was for an unknown startup which today I still don’t remember the name. However, what still stands out to this day is the question I was asked: do you know what continuous integration is? I was clueless, and I openly admitted to being so. And I vaguely remember the developer’s brief explanation of continuous integration (CI).

Needless to say I didn’t land the job. Several months later I would find myself working in a Software Developer In Test role. How could we embedded software quality and improve software testability were the main questions that came with the role. I was forced to examine CI and how teams where practicing CI, something that many developers and product managers still take for granted.

In CI, integration refers to the practice of combining source code changes with the main or master version of the source code. The main version of the source code is often referred to as the mainline, trunk, or master branch. Integrating frequently and ensuring the mainline is bug free and ready for deployment is what continuous is emphasizing.

The major components of traditional CI are:

1. Maintain a single source repository of code
2. Automate the source code builds. “Traditionally a build means compiling, linking, and all the additional stuff required to get a program to execute. A program may run, but that doesn’t mean it does the right thing. Modern statically typed languages can catch many bugs, but far more slip through that net.”
3. Run automate test. Test must be extensive and give significant confidence to developers about the state of the code base.
4. All developers commit directly to the mainline (usually called the master or develop branch on projects that do branching)
5. Everyone commits to the mainline everyday. In other words you commit the code changes you are working on to the mainline every day.
6. Every code change to the mainline should be built and tested
7. IMMEDIATELY FIX BROKEN BUILDS OR FAILING TEST. This is vital to achieving the underlying goal of having a healthy code base, ready to be deployed at a moments notice
8. Keep the build and test feedback fast
9. Test in a production-like environment, which could involve automating deployment.
10. Make it easy for anyone to get the latest app executable
11. Everyone can see the state of the mainline. So if the latest build has failed on the mainline, developers will not update their work by merging with the mainline. Communicating the state of the mainline is a critical part of CI.

Key things to note in traditional CI:

1. There are no feature branches. This is the result of working directly out of trunk or the mainline. The project is initially cloned and the developer start working on the mainline on their local machine.
2. There are no code reviews, but peer-programming can be used to act as a form of built-in code review.
3. Frequent integration is key, as mentioned in point #5 above. The code that a developer is working on is committed often (at least every day, if not several times per day).
4. In addition to point #6 above, builds and automate tests feedback should be quick, giving developers quick status updates on the source code health.

Often In Real Life

The typical way development teams approach their CI from what I’ve seen is by using feature branches. There is no development directly out of the mainline (trunk). This results in developers merging their code into the mainline via a merge or pull request as oppose to directly pushing code to the mainline. Feature branches are seen by many as short-lived branches but really they stick around for the duration of the sprint. Sprint cycles are commonly one or two weeks long. My understanding of short-lived is a day or two. Outside of this length of time, the savvy developer is rebasing their branch daily to safe guard their branch’s build integrity. Basically they pull in any new changes that are on the mainline without creating merge commits that would otherwise disrupt their branch’s commit history. The overall idea of short-lived branches is that they will be merged quickly to prevent integration issues, the branch is developed for a specific version, and it tracks all the new changes or commits related to the code in the branch.

Long-lived branches refer to branches that will house several release versions of the code base. Integration is not a concern is this case. The branch last for a long time and code housed in it diverges from other branches as time goes by. Teams may use a long-lived branch (besides mainline branches like develop or master) to build significant feature sets. Think epic branch.

Code reviews are used over pair-programming, some companies do both I know, but a sole code review process seems to dominate.

Here’s a Close 2nd

For teams that do not develop directly from trunk and use feature branches with code reviews on pull request, I’ve seen that the popular approach is to have two mainlines (develop and master). Having the git strategy of two mainlines, develop and master, supports creating feature and release branches from the develop branch, while creating hotfix branches from the master branch. Tagging is used to mark various versions of the application. Using the Git flow approach via Source Tree seems to be a developer sweet spot.

There is no substitute for test automation. It should be extensive and fast. Providing fast feedback from a reliable test suite is critical in monitoring the state of the code base on every commit to the mainline. Githooks are a developer’s best friend but unfortunately they are often under used. They are powerful and can guard against bad code being pushed to the repo. By using git hooks we can ensure developers run and pass test before they push their code. In a perfect world only the test pertaining to the code changes on the feature branch would run.

Any changes in the mainline trigger a build and run the entire test suite and this can include user interface test as well.

If a build fails on the mainline the development team will be alerted immediately and should resolve the issue immediately. Possible notification methods to alert the team are:

1. Slack; send a direct message to team members. Inform them that build has failed on the mainline and no code should be pushed to or pulled from the mainline.
2. Email doing the same thing as above

The ideal scenario on build and test failures on the mainline would be locking the mainline, thus preventing developers from pushing or merging code into a code base that has become unstable.

Strongly consider embracing the practice of rebasing and squashing commits. Rebase and rebase often when working in a feature branch. Repository settings can support this approach. For example, the Gitlab setting for merge method could be Merge commit with semi-linear history or Fast-forward merge:

Gitlab merge method options

Why rebase:

Besides being tidy and logical, a linear history comes in handy when:

Looking at the history. A non-linear history can be very hard to follow — sometimes to the point that the history is just incomprehensible.

Backtracking changes. For instance: “Did feature A get introduced before or after bugfix B?”.

Tracking down bugs. Git has a very neat function called Git bisect, which can be used to quickly find which commit introduced a bug or regression. However, with a non-linear history, Git bisect becomes hard or even impossible to use.

Reverting changes. Say that you found a commit that caused a regression, or you want to remove a feature that was not supposed to go out in a specific release. With some luck (if your code has not changed too much) you can simply revert the unwanted commit(s) using Git revert. However, if you have a non-linear history, perhaps with lots of cross-branch merges, this will be significantly harder.

There are probably another handful of situations in which a linear history is very valuable, depending on how you use Git.

Point is: The less linear your history is, the less valuable it is.

from Marcus Geelnard’s Bits ’n’ Bites

Do What Makes the Most Sense

The use of an epic long-lived branch is a major departure from conventional or traditional CI. Teams must do what they feel makes the most sense in the context of the project or product goals. What’s of paramount importance is that software developers and engineers understand what continuous integration is and where it came from. They should understand their departure from the original model, the consequences of such a departure, and be willing to create and support a team culture that ensures integrity of their main development branch on every code commit.

Striving to implement the conventional approach to achieving CI may not be suitable, but the methods described above can bring you to a close 2nd. This will lay the foundation for achieving continuous deployment.