What Does Our Test Case Management System Look Like?
If you conduct manual testing, then your company likely has a test case management system (TMS). I want to tell you a little about our one.
But first, let me say a few words about our team. We are the Quality Assurance Unit at Avito. You can read more about our company here.
Our unit is responsible for making tools for testing, both automated and manual ones. Also, we act as a center of excellence in testing. We help other teams to build and maintain their testing processes.
We have gained a lot of experience in managing it. And now we are ready to share it with the world, piece by piece.
In this article, I explain how we manage our test cases. Later I want to tell you about each part of the automated segment within our testing procedure I have ever touched. So, keep reading and stay tuned!
Test Case Management System
First of all, we have a handcrafted TMS. We call it Cthulhu. The name has come from “QA Tool”. And it has some features that distinguish it from any others:
- Branching model, like a lightweight Git.
- API for managing test cases and checking autotest status.
- Test cases can be linked to other test cases as a precondition (now it is a deprecated feature but it was nice).
- Infinite feature trees.
Branching Model
I should tell a little bit more about this interesting model. I had heard from our testers and developers that this is a really exciting feature that they never have seen in other TMSs.
All your test cases are in the master branch. The master branch should represent a production state. When a team is working on some problem, a tester creates a new branch in TMS that represents the state of this problem. Then they update or create some test cases. And after all, when the problem is worked out, the branch is merged into the master. Now we have a master with an actual test model.
A little about branching under the hood
There are no extraordinary technical details in branching. While creating a new branch, we do not copy all the test cases we have to the new branch. In a branch, we only store modified test cases. While fetching test cases from the branch, we fetch «all modified test cases and all test cases from the master» and then join them. Through this process, our database is not expanded with unusable data. And as a result, we get a free awesome feature: auto-merging master into a branch. While working on your branch, you can see how the test model for the entire project is changing over time.
But there we have a minor conceptual bug — we do not handle any merging conflicts. The last is the truth. It was a great challenge for us a few years ago. But now Avito that has become a cross-functional structure, and every team owns only its own part within the test model. This issue has gone away with our old functional structure. So now, finally, it is not a bug anymore 😅. Ok, well, it’s a negligible bug and nobody cares about it.
API For Automation
In Avito, we dislike manual work, so we want to automate everything. We use the TMS API for two main reasons:
- To set autotest passing status.
- To manage test cases via API.
Let me say a few words about autotests. A huge part of test cases has a number of associated autotests. Once an autotest with an associated test case has passed, all relevant information is sent to the TMS. This process sets certain autotest passing status, and it means that the test case should be marked as «automated», which performs automatically.
If you are curious about managing test cases via API — keep reading through the article 😉.
Infinite Feature Trees
We really love this feature. Most of TMS have a folder structure. Sometimes it can be only two-layer deep. With our feature trees, we can represent any test model with endless nesting. And a test case belongs not just to one feature, like files in folders. It can belong to several of them.
Also, it may sound strange, but… we have more than one tree. Here they are:
- The global feature tree — it represents a standardized tree of the entire project for all teams. Only particular members can edit it. But any tester in the company is familiar with the entire tree.
- Team feature tree — also knowns as a «unit test model». It represents a test model for the teams’ part of the product in a convenient way. The team can only edit their own parts of this tree in any way they want.
- API methods tree — this tree is only available for platforms with public API. It represents, hmm… API structure of the product.
A Few Small Tips that Help Us
First of all — divide and conquer. We have a lot of projects. Each project is divided into platforms. Some projects have a great number of test cases. For example, «Avito::Web», that represents our desktop site, has more than 4000 test cases. The average size of the platform is about 1000 test cases. But some platforms have just a few dozens of them.
The second one is Agile Testing. This is a big thing. It includes the test pyramid building. And, after all, the entire team is responsible for the quality. You cannot say «I am just doing my work, it’s not my problem». The whole team is involved in testing, in maintaining quality.
Plans to Seize The World
Our mission now is:
Each test manager should be able to see the entire test model of their part within the product in order to make the right decisions.
And our current goal is:
Each assertion we have in the code MUST be in our TMS.
It means that we are going to handle huge projects with about tens of thousand test cases in each one. But it is nothing in terms of Avito loads. Then we just keep calm and gathering test cases 🙃.
And in order to gather all the assertions, currently we are building something, that we call «The Tentacle».
The Tentacle
Our testers and developers do not want to do double work. They just want to write autotests and see how test cases «automagically» appear and match onto their feature trees.
We have a standardized representation of the test case in our TMS. But test cases may be stored in any place, even in Google Docs if somebody wants!
The Tentacle is a simple system — it grabs a test case from the outside (mostly from autotests), converts it into our internal representation and sends it to the TMS. That’s it. The TMS itself doesn’t care about the external world.
One complicated thing with The Tentacle is how to match an autotest with features. We just don’t care about it yet — we just mark our test cases with all feature IDs that they belong to.
New Test Plans
What is wrong with the current test plan system? I didn’t tell you anything about it. Because it’s just not suitable for the majority of critical cases. With the current test plan system, we can just add test cases to the test plan, assign them to users, click «Success» or «Fail» in each test case and… sadly, but nothing more.
Well, in this quarter or early next one we are going to completely rewrite our test plan system. It will get the following features:
- Test device management. We often test our product on different devices. But we have no tools to manage these devices. Each team deals with it as they can. We want to centralize this process. With this feature, each unit will be able to create its own pool of devices to use them in the next feature.
- Assigning test case to more than one device. And each team member mostly will have one device. With an option to have multiple devices. In Agile Testing world, the whole team is involved in testing. So we can speed up regression testing here. Today we do not have a completely clear process for it. It is possible to click «Success» or «Fail» in one item of the test plan only when it passes on each device. It just looks odd.
Afterword
I don’t have to say that we are planning to open source our TMS as soon as possible. At least we are so excited to share our ideas concerning automated testing and test management. Because now our TMS is so tightly integrated into our infrastructure.
If you have any questions, don’t hesitate to ask! File a comment, share your story, whatever you want! I’m glad to learn from your ideas, and I want to share more of ours.
Also, you can find me here or anywhere on the Internet with Enapiuz nickname.
