Let’s run only the tests that matters.. but who really cares?

The feature-branch Git workflow consist of creating a branch with a very small contribution and review a patch(i.e pull request) before merging into the master branch. However, during the review process, many automatic checks are usually enabled - and one of them runs tests. Automatic checks are evaluated for each new commit we apply to address the review comments.

Consequently, the automatic checks affect the time to merge your pull request, and thus, in your productivity.

Fortunately, many organizations are moving to a microservice architecture, which are small projects with a very specific responsibility, which means that automatic checks are quite fast to evaluate — (usually take between 10 to 20 minutes).

However, in some big organizations (e.g Google or Microsoft) it is quite common to find mono-repos. A mono-repo is a repository which embeds all the organization repositories/projects. This is quite common in big companies because you easily know the impact of shared libraries changes, and more importantly, they are always up-to-date. In other words, the lack of development tools (and I would say communication tools/strategy) have provoked that situation.

Big organizations with mono-repos have designed engineering teams that help to apply incremental evaluations/process over mono-repos — e.g modifying the Git internals to have faster operations. One working area is to evaluate the tests that are closely related to a pull request: it reduces the amount of resources that your Continuous Integration(CI) tool needs and the amount of time to wait until merging a pull request. This is also called Test Impact Analysis.

According my last searches in Google, it seems that there is not an widely accepted tool to apply this approach in any Java/Scala/Kotlin (JVM based) project — and I decided make a try, because it could produce a huge impact, isn’t it?

This is my idea: https://github.com/rpau/junit4git. The implementation quite simple:

First of all, the system(junit4git) needs to generate a test impact report which contains which classes are loaded by which test. For example:

[
  {
    "test": "CalculatorTest",
    "method": "testSum",
    "classes": [
      "Calculator"
     ]
  }
  ...
]

This is implemented using a Junit listener, which is a component that is notified every time a test is going to be started or has finished. This listener sends this information to a Java agent that is dynamically started via HTTP calls.

Java agents are components to instrument code, and thus the system can be notified every time an object is created. It means that the system knows the test that is being runned and the classes that have been instantiated by this test, which is the information we need for our test impact report.

After generating the test impact report for the master branch, if we decide to start to apply changes, the system can infer the tests that are not affected by our changes. Before running the application tests, if the system annotates those tests with the ignore annotations (at bytecode level), then we will be only running the tests that are affected by our changes.

mvn tests -- or gradle check
.
.
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 4

However, if you are familiar with automatic code coverage checks (e.g codecov.io), you already know that these systems compare the pull request code coverage reports with the master reports. Trying to reduce the execution build time exclusively running the tests that are related to your changes, means that your coverage report is incomplete. Then, there are two possible scenarios:

• Argue that code coverage tools needs to offer an “incremental” mode. If this is the case..keep waiting, because it probably means that these big companies have built their own “custom” coverage analysis and it is not yet available for the mere mortals.
• Argue that this policy is useful to test your code before pushing.. In that case, you probably know which tests do you want to manually execute from your IDE.

Which is your opinion?