You are merging untested code

The state-of-the-art practice on collaborative platforms like GitHub, is to run a suite of tests each time a developer sends a pull request. Each time a developer wants to merge something, their pull request has to pass the continuous integration system that validates everything, from unit testing to code style compliance.

When a pull request looks like this:

You know it’s ready to merge. That’s where Mergify can help you and merge the pull request for you, as we’ve seen last time.

However, if you think that merging this pull request will not break your software because it’s all green, you’re wrong.

To understand why, imagine 2 pull requests, A and B:

  • A adds a feature that paints a yellow emoji 😬 on the screen. The code is correct, the tests pass , and the developers approved it. This pull request is ready to merge.
  • B adds a different feature, which makes sure that nothing yellow is ever printed on the screen. The software does not print anything yellow for now, so the tests pass, and the developers approved it. This pull request is ready to merge.

For Git, A and B does not conflict: they modify different files, so they are mergeable together with the target branch. Here’s the B pull request continuous integration status:

The B pull request is mergeable

One of the developers clicks on the button to merge A. So this is the state of the continuous integration system for B now:

The B pull request is marked as passing and is ready to be merged, even if master evolved.

Another developer clicks on the button to merge B, and here is what you get:

Now, when running on the target branch, the test suite fails: the software is unhappy because there’s something yellow printed on the screen. And yet, everything was shiny green.

Bummer. 😬

Some people will object that in this case, only A or B should have been merged and that the problem lies with the developer. While it might be true, it’s also very naive: transpose this example to large and complex software projects with tens of developers, and it’s more than likely that this kind of issue will happen.

Solving concurrent and incompatible merge

There’s a simple way of solving this issue: disallowing any merge if the pull request is not strictly up to date with its target branch.

How does that work? Well, starting with the same conditions, one of the pull requests is going to be merged, let’s say A.

Now, B becomes un-mergeable. Why? Because it is not up to date with its target branch anymore. The target branch now contains A, and B has never been tested with A. Therefore, it needs to be updated with the new code that has been merged in the target branch. Once this is done, B will fail the continuous integration system, making developers aware of the issue before any broken code is merged into the mainline tree.

As you might guess, this workflow, which we name strict, is available when using Mergify. Once enabled, Mergify handles everything that’s described here, from picking a pull request to merge to updating the other pull requests with their target branch, to make sure they won’t break their target branch once merged. More details are available in our documentation.

Once you understand this problem, you’ll see the flaw in many continuous integration systems, unfortunately.

Except for the ones using Mergify. 😬