AI in QA: keep TestRail up-to-date

How to use AI in Software Testing? I don’t know. I just want to start using it more than chatting.

This article is not about the problem statement but more about using GPT in practice.

I decided to start with something simple, something that requires QA team attention but due to the workload it is always “not a priority”.

We use TestRail to keep our test scenarios in the style of Given/When/Then (we don’t use Cucumber). We have a custom integration between a test automation framework (TAF) and a TestRail. When we run the tests via CI (we use GitHub Actions), it creates a TestRun, updates test statuses and publishes the links to Slack.

Good question: Why do we need a TestRail? This is out of the scope of this article :)

Our QA process is simple: each User Story has a bunch of BDD use cases that we refine and convert into E2E automated tests.

The test base is growing and when something is changing in the system, we just “adapt” automated spec and don’t have time or desire to update the TestRail use case. So with time, it is a clear problem to keep TestRail up-to-date.

It was one of the ideas where I could use ChatGPT: to keep TestRail up to date.

The plan is simple:

1. Get automated test content.
2. Generate a Given/When/Then use case based on it.
3. Update TestRail case.

I decided to start from the second point by using ChatGPT as a playground. I created my GPT with instructions (prompt) to generate a use case based on the provided automated spec.

I will say right away that the main task is to write the correct prompt. It took me ~20 iterations to polish the prompt and now I’m seeing >90% results that I’m happy with.

The next task is point 3: update the TestRail case. For this purpose, I created a GPT Action.

First, I found a TestRail API request for updating the case: https://support.testrail.com/hc/en-us/articles/7077292642580-Cases#updatecase. With a little experimentation via Postman, I have found the POST body request that I need to use:

{
    "custom_testrail_bdd_scenario": [
        {
            "content": "..test scenario here..."
        }
    ],
    "template_id": 4
}   

Then I asked ActionsGPT to help me generate openapi spec https://chat.openai.com/share/03b063e0-04cb-43d8-b245-3cfde56170dd and added it to the “Schema”.

[screen with comment]

The trick was with a TestRail API authentication. It suggested using username:password or username:api_token, but I had to additionally encode it (base64):

Finally, the prompt for my GPT was simple: “Make updateCase with case id: 1” and a piece of code of the automated test.

This simple experiment shows where to move further.

1. Use TestRail and OpenAI API inside my test automation framework to automate these steps.
2. Update TestRail case as a part of Pull Request.

So why do I need the TestRail? Initially, I did not see a value of it when all tests are automated but now it can be used in many ways, for example, as a knowledge base for GPT, but about this later.