MiQ’s Test Management Tool — Heimdall
Automated Tests scripts written worked peacefully locally. Went a further way and did a Jenkins setup, now tests were running seamlessly on Jenkins. For better reporting and test case descriptions, the Cucumber framework was used. This eased our task and we started getting a good result on a daily basis on email. But there was something that was still missing.
- Each time our Product Manager or Developer asked for the result of a particular execution (date or build) we had to dig into emails to fetch them that
- If they asked us to execute a few tests (sanity tests or some feature tests) then we had to go back to Jenkins and execute the same for them.
- Also if someone asked us to share certain test cases we had to export them out from Jira and then send them the same on email.
So we at MiQ thought to solve this problem. We wanted to create a tool which addressed the below problems :
- Single repository of all the test cases
- Whenever a new test case is added, it should automatically be pushed to the above repository
- User should be able to execute the tests from the same repository
- Results should be available on the repository
- And the most important thing, the results should be easily shareable which means that you don’t have to send an email or put any extra effort.
Ge’ez…these were lots of things to consider.
So here at MiQ, we developed a tool called Heimdall, which is a Test management and Reporting tool built on top of Google Sheets. It’s a one-stop-shop for test repository, test scripts execution, execution results and sharing results. Since it is based on Google Sheet you can build your own execution/build results charts and tables.
Why Google Sheet?
- Free web-based tool
- Easy data representation
- Ease of collaboration
- Google scripts require minimal JS knowledge
So we used Google scripts as our repository, created an Add-On for test execution, and to fetch test execution results. Used S3 as storage for results (results store also available in the sheet itself) and Jenkins for execution. We used Java for processing the results before pushing it to S3. We also support publishing results to Slack.
Minimum Requirements:
- Exported Test Cases on Google sheet ( you can use Mobility Stream Plugin as well)
- Jenkins Job (To execute Test cases)
- Cucumber or Karate framework support
- Tagged scenarios in feature files (Helps to Generate Reports at Tag level)
- Slack for notifications
Steps to set it up:
- Export Test cases on a Google sheet. Make sure you have a STATUS column as the last column.
- Now you have to create a Jenkins job with the same name as in your sheet name (Google Document name can be anything). This helps us to map Jenkins’s job with tests in google sheets as one Google document can have multiple types of test cases depending upon project or services.
- Install the Heimdall Add-on using the below link
- Steps to configure Heimdall Add-on:
- Select the Add-ons menu from the Google sheet → Heimdall → Configurations
- Provide Jenkins Token Name, Token Id, Domain, Job Category
- Provide S3 Access Key and Secret Key
- Add Heimdall maven dependency link
- Once dependencies are downloaded then make linked changes for Cucumber project and Karate Project
Test execution:
There are two ways to do test execution.
- Via Google sheet
- Via Jenkins job
Test execution results:
Test execution results are also available at 2 places:
- On Slack as a notification
2. Google sheet from where you executed the test cases. For this, you need to go back to the Heimdall add-on and select Update Build Info for updating the results on a google sheet.
The execution result at both the places Slack and Google Sheet will have details of the test build executed.
The error results in the Google sheet will give you elaborated detail to check why the test cases failed.
We have also added some color formatting for you to distinguish between builds.
- Solid green color if all the test cases passed
- Pink color if the number of failed test cases is greater than the passed test cases
- Red color if all the test cases failed
- Yellow color if the number of passed test cases is greater than the failed test cases
Now the advantage of having everything on Google sheet is that you can share the Google sheet with anyone whom you want to and now that person from the same sheet can see the execution results daily and can even execute scripts from there itself. Another advantage of having it on Google sheet is that you can create pivot tables and charts to further tell the stability of a build.
We have also open-sourced this tool on GitHub: https://github.com/MediaIQ/heimdall
Summary:
Taking advantage of Google sheet, google scripts and Java we have created an in-house tool Heimdall which have the following features:
- Single repository of all the test cases
- Anyone with access to Google sheet can execute the tests and check result
- Sharing of the result is easy
- The execution results are available in Slack and Google sheet
