Generating a Smoke Test Suite

Hi! My name is Stanislav and I’m a technical lead in the QA Automation team at Wrike. In this article I’ll tell you how we managed to run less tests and how you can automate some of your work.

How not to run extra tests

At Wrike we have more than 30,000 tests. It’s expensive to run all of them on every developer branch.

Everyone knows how to run tests, but not everyone knows how not to run extra tests. Imagine you have a lot of tests and you want to be able to run only some of them.

There are at least two possible solutions.

• By markup. We can run a set of tests marked for the product feature that’s being tested. You can find more information on the test markup we use at Wrike in this video.

But sometimes developers impact other parts of the application, want to make sure that everything is working properly, or don’t have a product markup for tests. Then we have a second option.

• Smoke suite. The second option is smoke testing. We pick the most critical tests in the product and build a test suite that checks an application’s basic functionality.

Many people pick these tests manually, but we went further.

Why we decided to automate the process of smoke suite updates

Our product is constantly evolving, which means we continuously check and update the smoke suite to match the product. It can be time-consuming due to the size of the product and also involves several teams to decide which tests to pick. You should consider if tests are still relevant, whether they’re stable enough, how long each test runs, how critical that test is, and so on. And to change the smoke suite you have to change a code. That’s a lot of manual labor, isn’t it?

That’s why we decided to automate the process of our smoke suite updates to increase the frequency of testing with less effort.

The algorithm for building a smoke suite

In our project structure there are several layers:

• Page elements represent product frontend structure.
• Step methods use page elements to perform some actions or checks on a web page. They’re marked with an @Step annotation.
• Tests use step methods to implement test scenarios. They’re marked with an @Test annotation.

We decided to pick tests for the smoke suite in such a way that they’ll call all the steps we have at least once. With this approach we can be assured that we push every button, fill every textfield, and visit every page we have.

Let’s try to build a smoke suite. It’s obvious that by picking one test we’ll have all the steps covered.

But what if you have several thousands of tests and a very complicated structure like in the image below?

Imagine choosing smoke tests among this web of usages. We do it using our own IntelliJ IDEA plugin. It can parse a project’s code and make the required changes. Our plugin comes in handy here. It can pick the most useful tests to fulfil our requirements.

So how does it work?

It chooses the least used step and picks the test that contains the most steps:

Then we mark all the steps from this test as used and repeat the process with the remaining steps:

Pick the test, mark the steps, and continue until every step is covered:

Voila! We picked only three tests and covered all the steps. After that we simply mark chosen tests in the code as a part of a smoke suite. For our project it took only 1,000 tests from 30,000 to get full coverage.

How to use an IDEA plugin

The time it takes for the IntelliJ IDEA plugin to update the whole test suite is about 10 minutes for our giant project, so we can do it as often as we need to keep up with the product development.

It works so fast because IDEA builds indexes containing Program Structure Interface elements (or PSI in short). Plugins extend its functionality and use PSI to work with code lightning fast.

If you’re not familiar with the IDEA plugins concept and how it works I suggest you watch a video about it.

I’ll show you how to use this plugin. We have it on GitHub so you can download it and modify it for your needs.

The plugin tool window contains two tabs: configuration and actions.

On the Config tab you need to specify the fully qualified names of annotations that you use in your project. We already have presets for JUnit 4 and 5. The plugin will use these annotations to parse code and mark up your tests.

Step annotation is the annotation with which you mark your step methods.

Test annotation and Before each test are the annotations from the test framework. Tests markup is the annotation to combine tests into suites. And finally, Tests markup value is where you add the name of the test suite.

The main buttons are on the Actions tab. Firstly, the plugin needs to parse your code and build a map of step usage in tests.

You’ll see how many steps it found and the maximum coverage. Don’t be surprised if you see less than 100% maximum possible coverage like in our example. Some steps aren’t used so you can’t cover them with current tests.

Click Pick smoke tests for the algorithm to pick a set of tests to retrieve the maximum coverage of steps. You’ll see how many tests have been picked and the final coverage.

You can reduce the number of tests by ignoring the steps that are used in fewer tests than specified. The number of steps usage can indirectly indicate the importance of this step.

In our example we received almost two times fewer tests by ignoring the steps with one or zero usages. But the coverage is still decent.

When you’re satisfied with the results, you can finally remove an old markup and add a new one. Then all you need to do is commit changes and merge it to the “master” branch.

Make your own IntelliJ IDEA plugin for smoke test markups. Or download a prebuilt one.

You can easily automate your recurrent and time-consuming jobs like this using the IDEA plugin or other tools you have. The only thing you need is to think about what you can automate and how to do it. After all, we’re called an automation team for a reason!