Test Markup in QA Automation

Hi! My name is Ivan and I’m a QA automation technical leader at Wrike. In this article, I’ll discuss how test markup works in QA automation, what problems it solves, and how you can implement it in your projects.

What is the problem?

Imagine that you’re developing a complex web application with many views. The application also needs to be deployed every day. To make this possible, you need to have a powerful QA automation framework with many UI tests that will help you to find bugs in your product.

In our case, this complex application is Wrike. Wrike has a lot of views for managing your projects and tasks, including task views, custom workflows, proofing, reports, timelines, Gantt Chart, etc.

There are about 30 Scrum teams that develop these views and make internal changes within the product, and we usually deploy about 30 new tasks every day. That is why we have high requirements for QA automation and our test project has more than 30K Selenium tests and more than 12K API tests. The run of all tests takes about two hours, which is quite long and means we can’t afford to use this “test suit” on its own. Therefore, it’s very important to use cheaper stages where we can check the product.

The first stage is unit testing. In my opinion, this stage is the clearest one. We can run all unit tests after each change because they are fast, stable, repeatable, and predictable.

The second stage is feature testing. During this phase, we usually check some features in isolation and run some API and Selenium tests. But here, we get a question: What tests should we run on this stage? To answer this, we use test markup.

Test markup

Test markup is a set of rules and instruments that allow you to mark the tests. We use the Allure framework for this purpose. You can find more information about our integration with Allure in this article .

How does it work?

1. Mark up the tests with annotations: If we’re talking about Java, we use the annotations in the Allure framework. These are “Epic”, “Feature”, and “Story”.

For example, the Story annotation with the “Login” value means that the test checks the login part of our product. Besides the “Epic”, “Feature”, and “Story” annotations, Allure has many other useful ones, like “DisplayName” that provides a human, readable name of our autotest.

2. Run tests: After marking up the tests, we need to run them.

3. Generate an Allure Report: In the report, we can find a lot of useful information like status, display name, and, in this case, story with a login value.

So, when we have some annotations in the Allure framework, we can use them for marking up our tests and this information can be reflected in a generated Allure report. Let’s move on and discuss the rules of test markup.

Rules of test markup

Imagine there’s an application and specific tests that check only this application.

Often, each frontend view consists of components. We can then find the necessary suit that will check the exact component. In our example, we’ll split the tests into some suits. How do we know which suit checks which application or component? We can use Allure annotations to answer this.

Let’s imagine that this application is called “Task View”. We can reflect this information by using the “Epic” annotation from the Allure framework and adding this annotation to our autotest.

Now we have the @Epic (“Task view”).

Then, we find out that this test checks only the upper component of the Task view application called “Header.”

We can reflect this information in the autotest project and add a “Feature” annotation with a (“Header”) value.

Finally, we understand that the test checks only the small component inside the header called “Status Suggest.”

We can use this information and add a “Story” annotation with a “Status Suggest” value. So far, our test has this set of annotations:

“Epic”, “Feature”, and “Story’ are like coordinates of our test and it would be great if we could use these coordinates to run the test.

How to run tests using Allure annotations

To run tests using Allure annotations, we developed adapters for each famous test framework from the Java world — JUnit 4, JUnit 5, and TestNG.

These adapters use a special language for test running:

• Operators: !, |, &. We can use exclude, logical “or”, or logical “and” operators.
• Set of tests for story: {story:name_of_story}. {story:Task suggest} — this string matches a set of tests where each test has @Story annotation with a “Task suggest” value.

The same logic applies for the ”Feature” and ”Epic” annotations:

• Set of tests for ”Feature”: {feature:name_of_feature}
• Set of tests for “Epic”: {epic:name_of_epic}

It’s possible to mix these sets and operators to make more complex test filters like “{epic:Task View}&{story:Task suggest}”. It makes sense when there are other tests with a “Task suggest” story but not in the “Task View” application. All information about these adaptors is available on our GitHub.

Let’s take a look at an example. We only want to run tests for the header.

In this case, we need to run tests that have @Feature with a “Header” value. For that purpose, we use the “{feature:Header}” test filter. Then, we will run only the necessary tests and get the results.

Bonus: Test coverage map

Imagine that all our webtests have “Epic”, “Feature” and “Story” annotations. In this case, we will get a little bonus. Let’s look at the “Behaviour” tab in the Allure report.

Here, we can see a tree where the first level is ”Epic”, second is ”Feature” and the last is “Story”. This provides us with a top-level understanding of the test coverage by product features. The picture can explain how many tests there are for each story and whether the application is properly covered by autotests.

To sum things up

Allure isn’t just a report. It can be used to improve transparency by using the “Epic”, “Feature”, and “Story” markups. You can use this markup to run special suits of tests by using our adapters and test runner, thereby optimising your time and resources.