You are merging untested code

Jul 2, 2018 · 3 min read

The state-of-the-art practice on collaborative platforms like GitHub, is to run a suite of tests each time a developer sends a pull request. Each time a developer wants to merge something, their pull request has to pass the continuous integration system that validates everything, from unit testing to code style compliance.

When a pull request looks like this:

Image for post
Image for post

You know it’s ready to merge. That’s where Mergify can help you and merge the pull request for you, as we’ve seen last time.

However, if you think that merging this pull request will not break your software because it’s all green, you’re wrong.

To understand why, imagine 2 pull requests, A and B:

  • A adds a feature that paints a yellow emoji 😬 on the screen. The code is correct, the tests pass , and the developers approved it. This pull request is ready to merge.
  • B adds a different feature, which makes sure that nothing yellow is ever printed on the screen. The software does not print anything yellow for now, so the tests pass, and the developers approved it. This pull request is ready to merge.

For Git, A and B does not conflict: they modify different files, so they are mergeable together with the target branch. Here’s the B pull request continuous integration status:

Image for post
Image for post
The B pull request is mergeable

One of the developers clicks on the button to merge A. So this is the state of the continuous integration system for B now:

Image for post
Image for post
The B pull request is marked as passing and is ready to be merged, even if master evolved.

Another developer clicks on the button to merge B, and here is what you get:

Image for post
Image for post

Now, when running on the target branch, the test suite fails: the software is unhappy because there’s something yellow printed on the screen. And yet, everything was shiny green.

Bummer. 😬

Some people will object that in this case, only A or B should have been merged and that the problem lies with the developer. While it might be true, it’s also very naive: transpose this example to large and complex software projects with tens of developers, and it’s more than likely that this kind of issue will happen.

Solving concurrent and incompatible merge

There’s a simple way of solving this issue: disallowing any merge if the pull request is not strictly up to date with its target branch.

How does that work? Well, starting with the same conditions, one of the pull requests is going to be merged, let’s say A.

Now, B becomes un-mergeable. Why? Because it is not up to date with its target branch anymore. The target branch now contains A, and B has never been tested with A. Therefore, it needs to be updated with the new code that has been merged in the target branch. Once this is done, B will fail the continuous integration system, making developers aware of the issue before any broken code is merged into the mainline tree.

As you might guess, this workflow, which we name strict, is available when using Mergify. Once enabled, Mergify handles everything that’s described here, from picking a pull request to merge to updating the other pull requests with their target branch, to make sure they won’t break their target branch once merged. More details are available in our documentation.

Once you understand this problem, you’ll see the flaw in many continuous integration systems, unfortunately.

Except for the ones using Mergify. 😬

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store