Autotesting Optimized: Intellij IDEA Plugins in Service of QA Automation

Hi there! I’m a QA automation engineer at Wrike, and I’d like to share how we optimized the code review process with 30,000+ autotests by using an IntelliJ IDEA plugin. I’ll cover the internal structure of the plugin and what problems it solves at Wrike. There’ll be a link to the Github repository with the plugin code at the end of this article. Feel free to integrate the plugin to your processes.

Autotests and deployment in Wrike

We write a lot of different Java code within the framework of our autotest project. We now have 30,000+ tests checking our product through Selenium, REST API, WebSocket, etc. All tests are split into many Maven modules in accordance with the frontend code structure. The project is actively changing. New releases come out up to three times per day, and, to maintain the quality of the code, we use a well-developed code review process. During the review, along with the quality, we also check whether the tests are working properly.

Though autotests are important, they’re useless without the product being tested. In general, adding new functionality to a product implies the presence of frontend and backend branches, and also a test branch in the autotest project. All branches are deployed simultaneously. Every day we merge dozens of branches into the main branch master during several deployments. We run tests before the deployment starts. Errors in the code of the autotest project inevitably lead to an undesired deployment delay.

To avoid such situations, we decided to add one action to our code review process. The author of the merge request should attach a link to the passed test in Teamcity, and the reviewer should follow the link and check whether the run has really passed.

What problems can emerge when running autotests?

The author of a merge request can face problems even during a simple task such as adding a link to a run. If a QA automation engineer (QAA) changes a lot of code in the project, it can be difficult to understand which tests to run to make sure that nothing fails without running all of them.

We have two mechanisms for running tests:

1. Run by groups (product markup) — Epic/Feature/Story
2. Run by identifiers (id) — any autotest has a unique number assigned, so you can initiate a run via a set of corresponding numbers.

In most cases, we opt for the group run (product markup) — specifying a group is faster than collecting autotest ids for the entire project. Judging by the changes in the code a QAA tries to figure out the autotests of which parts of the product were affected.

Further events can develop as follows:

Scenario 1: Fewer tests were run compared to what’s actually affected by the new code.

So here’s the first problem — the potential slowdown of the deployment process. After all, there’s a risk that a test was missed, and it stopped working due to changes in the code. It’ll show up during deployment and will eventually slow down the delivery of the product.

Scenario 2: The opposite — more tests were run.

And here’s the second problem — excessive workload for the testing infrastructure (which the company pays for). This increases the tests queue in Teamcity, making one wait longer for the results.

To solve these problems, we created Find Affected Tests — an IntelliJ IDEA plugin that quickly and reliably finds the ids of tests affected by changes in the code of the autotest project.

From business problems to implementation

Why use IntelliJ IDEA plugin, you may ask? Creating a tool that solves our problems required answering two questions:

• Where should I get the code changes from?
• How do I find the affected ids through the changes in the code?

The answer to the first question was obvious. We use Git as a VCS and through it we can get information about the state of the current branch.

The answer to the second question was PSI (Program Structure Interface). I chose this particular tool for several reasons:

1. We use IntelliJ IDEA as our IDE
2. The IntelliJ Platform SDK includes PSI, a handy tool for working with project code that’s used in IntelliJ IDEA
3. The functionality of IntelliJ IDEA can be extended via plugins.

The third reason defined the tool for detecting autotests ids — a plugin that’s built into the IDE and allows you to get a list of test ids that were affected during development.

Plugin structure

Let’s take a look at the plugin UI to imagine what a QAA sees before starting the id search:

This is how the simplified version posted on GitHub looks like

Everything’s intuitive — run a search, look at the result. The result consists of two parts: ids of the tests affected and a list of Maven modules in which these autotests are located (I’ll explain why we need this list later in the article).

The internal structure of the plugin consists of two main modules:

• The Git module, which receives information about the current branch and converts it to a format PSI can understand
• The PSI module, which looks for test ids affected by changes in the current branch (based on data from the Git module)

The interaction of the plugin’s UI with modules generally looks something like this:

AffectedCodeUnit and DisplayedData are classes for transferring data between modules

Git module

I collected all commits from the current branch that aren’t in the main branch yet, since changes in the code (compared to the master version) were important to us. Each commit includes changes to some files.

For each file, the following data is collected:

• File name, module name, and file path relative to the project’s root folder. This makes it possible to further simplify the search for a file through the PSI mechanisms.
• Numbers of the modified code lines in the file. This allows you to understand what exactly has changed in the structure of the Java code.
• Was the file deleted or created from scratch? This helps optimize work with it in the future and avoid errors in some situations (e.g., avoid accessing a deleted file via PSI).

All this information is listed in the git diff command output. An example of the result looks like this:

You can notice the path to the modified file in the command output right away. The data on the line numbers of the changed code is found in the form of “@@ -10 +10.2 @@” output lines. You can read more about the output from git diff at Stackoverflow.

How does the interaction with Git work in the plugin code? I tried to look up the built-in IntelliJ IDEA mechanisms and learned that there’s a plugin for Git in IntelliJ IDEA — git4Idea. In fact, it’s a GUI for standard project operations. To access Git, our plugin uses the interfaces from the git4Idea code.

As a result, we got the following scheme:

A diff is requested through the GitBranch class, and then the diff is sent to the DiffParser. The output lists objects of the AffectedCodeUnit class (they contain information about the modified files with the code). I’ll explain this list more in detail below.

PSI module

Now we need to apply the information about the changed files to get the autotest ids. You need to take a closer look at the PSI mechanisms to find the elements of Java code through the numbers of the changed lines.

How does IntelliJ IDEA work with a random project? For PSI, any file is converted into a tree of individual elements. Each element implements the PsiElement interface and is a structural unit of code for a specific programming language. It can be an entire class or method or the closing brace of a small block of code.

In our task, the Java code element is the PsiElement object. So we can pose the question another way: How do we find PsiElement objects when all we know are numbers of the lines of code that changed? The PsiFile interface (the representation of a file with code as an object) inherited the findElementAt method, which is able to find PsiElement by an offset relative to the beginning of the text representation of a file.

A new problem popped up, though: Each line in the file potentially contains a lot of characters, and one line number can get us many different PsiElement objects.

How do you pick the right PsiElement object? IntelliJ IDEA maps Java code into the tree, which can have a huge number of nodes. A particular node can describe a minor piece of code. In our case, nodes of the following specific types are important: PsiComment, PsiMethod, PsiField, and PsiAnnotation.

Here’s why:

String parsedText = parser.parse(“Text to parse”);

In this snippet, you can change different parts of the code: the name of the parsedText variable, the input parameter of the parse method, the code after the assignment operator (you can invoke another method instead of parse), etc. There are many types of changes, but it’s important for the logic of the plugin that a code fragment can be inside a method, as in this example:

public String parseWithPrefix(Parser parser, String prefix) {
    String parsedText = parser.parse(“Text to parse”);
    return prefix + parsedText;
}

In such a situation, there’s no point in trying to understand what’s changed in the line of code with the String parsedText declaration. The only thing important for us is that the parseWithPrefix method has changed. Thus, we don’t need to be precise when looking for PsiElement objects for a modified line of code. So I decided to take the character in the middle of the line as modified and look for the PsiElement attached to it. Such procedure made it possible to get a list of objects affected by PsiElement and use them for searching the ids of autotests.

Yes, you can make up some crazy examples of Java code formatting that requires more conditions in searching for a PsiElement object:

public String parseWithPrefix(Parser parser, String prefix) {
String parsedText = parser.parse(“Text to parse”);
return prefix + parsedText; } private Parser customParser = new Parser();

Changes to the last line can affect both the parseWithPrefix method and the customParser field. But we have code analysis mechanisms that won’t allow this to make it to the master.

How does the basic algorithm for obtaining autotest ids for a set of PsiElement objects work? First, you need to be able to get PsiElement’s usages in the code from the object itself. This can be done with the PsiReference interface, which describes the relationship between the declaration of a code element and its usage.

Now here’s a brief description of the algorithm:

1. Get a list of PsiElement objects from the Git module.
2. Look for all of the PsiReference objects for each PsiElement object.
3. For each PsiReference object: Check for id and store it (if found).
4. For each PsiReference object: Look for its PsiElement object and recursively run the described procedure on it.

The procedure should be repeated as long as the PsiReference objects are found.

The algorithm for the purpose of searching the code for abstract information looks like this:

In our task, we need to find the autotest ids. Here’s how you can apply the flowchart above for this:

Algorithm optimization I ran into one problem while implementing the algorithm: It works with a PsiTree object that consists of a large number of nodes. Each node corresponds to elements of Java code. The default search engine for PsiReference objects can find rather minor elements of Java code, which will cause the algorithm to perform a lot of unnecessary operations while moving along the tree. This slows down the algorithm by a few minutes in cases where about a thousand tests or more are found.

Here’s a prime example — suppose the value of the wrikeName variable has changed:

public List<User> nonWrikeUsers(List<User> users) {
    String wrikeName = "Wrike, Inc.";
    return users.stream()
        .filter(user -> !wrikeName.equals(user.getCompanyName()))
        .collect(Collectors.toList());
}

Therefore, looking for wrikeName usages, we first come to invoking the equals method, then to negating the result of the invocation of the equals, followed by lambda inside filter call. But this long chain of steps can be replaced by searching for usages of the changed nonWrikeUsers method.

And then instead of four iterations of the algorithm for each node of the tree, we get only one:

The problem of unnecessary operations in the algorithm was solved by limiting the data types of PsiElement objects to PsiComment, PsiMethod, PsiField, and PsiAnnotation. It’s these PSI entities that contain all the relevant information for searching the ids of the affected autotests.

• A relevant info example: A field of some class was affected (an object of the PsiField type), and could have been used in some autotests.
• An irrelevant info example: A local variable declaration was affected (PsiLocalVariable type of object). Here, we only care about where this variable is (e.g., in the body of another method). Its value change is irrelevant.

I started the algorithm searching for the grandparent node with the required type of data. The search is executed via the getParentOfType method of the PsiTreeUtil class.

I implemented processing logic for each type of PsiElement object, e.g., a FieldProcessing class object is processed for the PsiField grandparent type, and a CommentProcessing class object — for PsiComment type.

The plugin code executes the suitable logic based on the input from the getParentOfType method.

In general, the working logic looks like this:

An example of the implementation of the logic for processing one of the types of PsiElement objects:

Summary of the main features of the PSI module:

1. The module works according to the recursive algorithm for searching autotest ids. In short, the idea of ​​the algorithm is to find ids for all modified PsiElement objects through the PsiReference mechanism.
2. The algorithm doesn’t work with all the nodes of the PsiTree object to avoid redundant iterations.

Additional features of the plugin

After we started using the plugin at work, colleagues sent several requests to improve it. This feedback helped improve the tool and optimize autotest runs.

Making the work with the Git module more flexible

It was possible to compare the current branch to the master in the initial version of the module (i.e., the last commit for the branch was taken). But sometimes this led to the needless work of the infrastructure — exactly what the plugin was supposed to deal with.

Example 1: QAA creates a merge request with one commit. It runs 1,000 tests that are affected by this commit. The reviewer leaves comments on the code, the author of the merge request makes edits. These edits now only affect 200 tests, but the plugin will offer to run 1,000 tests from the original run as well, since the first commit is taken into account. Now we got extra tests to run.

Solution: I added the ability to compare the local version of the branch with the remote version. This will only count commits that haven’t been pushed yet.

Example 2: QAA creates an integration branch to temporarily merge other branches in it. The code will eventually be pushed from the integration branch to the master. On code review, the branch set to merge into our integration branch will include a number of latest commits. But what if these commits have already been checked via the plugin before? We’re going to get extra tests in every run now.

Solution: I implemented functionality to compare the local version of a branch with an arbitrary branch (in general, with any commit). This allows you to account commits when working with multiple branches.

Implementing new optimization practices via plugin

Understanding the benefits of this improvement requires me to provide you some context. As I mentioned above, our project consists of a huge number of Maven modules. Our code was recursively compiled in all modules (even if only one module required). As the number of Maven modules in our project grew, autotests became quite time consuming. We encountered the problem of needless infrastructure work again.

We decided to change the startup script, and now we only rebuild manually specified modules. This allows us to reduce the compilation time from 20 to 3–4 minutes on average.

To find out which modules have the affected tests, I implemented the affected modules output in the plugin. To implement it — when getting the autotest id — you need to take the paths both to the file with the test in the operating system and relative to the project root folder. Knowing the directory structure of a typical Maven project, you can cut everything that’s unnecessary and leave part of the path with the names of the modules.

As a result, it helped to implement the new startup logic quickly and, therefore, reduce the queue time for each run and increase the infrastructure efficiency.

Further plans for development

The plugin goes online. We have automated the process affected test ids collection. However, QAA still has to perform a set of routine actions after that to initiate the run and further work with it:

1. Open the required build in Teamcity.
2. Trigger build with the required data.
3. Save a link to the run for results to be checked.

It all distracts from solving other tasks and leads to possible mistakes: initiating runs in the wrong Teamcity build, or errors in copying data for the run.

So the task emerged: to teach the plugin to initiate a run in Teamcity on its own. We have a special tool that accumulates most of the data related to CI/CD processes. Therefore, it’s easier for the plugin to send a run execution request to this tool (the link to the run will also be saved in it). The implementation of this logic will reduce the number of routine actions after writing the code and will make the process of running autotests more robust.

Multithreading. With a large number of changes, the plugin’s running time increases drastically, so we have plans for further optimization related to multithreading.

The idea is to have several threads analyzing different nodes of the Psi-tree simultaneously. To do this, you need to correctly organize the work with shared resources (e.g., with many already processed PsiElement objects). One of the possible solutions could be the Java Fork/Join Framework, since working with a tree implies recursiveness. But there are pitfalls: IntelliJ IDEA has its own thread pool, and you need to correctly embed new logic into it.

Some takeaways

How has the plugin influenced the problems it was designed to solve? Deployment time is a complex value influenced by a large number of factors. There are probably no metrics that can reflect the direct impact of a plugin on this value. Therefore, let’s get straight to the problem of running autotests during the code review process.

We have a build in Teamcity, in which we usually run autotests by hand. It allows us to initiate a run with Maven modules already specified. I chose this build as an example, because the author of the merge request attaches a link to it during the code review. Also, in this build, the runs are triggered manually, and the QAA can only get a set of ids for the run from the plugin (hardly anyone would consider gathering this set manually).

Here’s the graph with the total number of runs in this build:

In March 2020 (data point 03.2020) we officially announced the build, and in April (data point 04.2020) I added the Maven modules output to the plugin, in which the affected autotests are located. After that, the number of runs have multiplied. The next leap was in September, when we began using this build to restart autotests from our internal deployment tool.

The next graph shows that runs take place exactly by ids, and not by product markup. It shows the percentage of runs by tests id to total runs for this build:

Over the past six months, about 70–80% of autotest runs are executed with the id list specified: Colleagues are actively using the plugin for its intended purpose. In April, we added Maven module output to the plugin. The graph shows an increase of runs with ids from 50% to 75% in a few months.

Are id runs really faster than product markup runs? The average run time statistics for this build says yes. The “id” runs are approximately three times faster than the product markup runs: 8–10 minutes vs. 25–30 minutes.

The data on the charts proves that the plugin is being actively used. It means more Teamcity builds are using a more reliable autotest selection algorithm instead of product markup.

Leave comments with your successful (or not so successful) automation experiences.

If you’re interested in this plugin (and you have thoughts on how you can implement it), feel free to use the simplified code of its base version. There’s also a dummy project for the plugin testing. I’d be happy to hear comments and suggestions regarding the architecture of the code.

All the best and more effective optimizations to you!