Chat GPT QA: Revolutionizing Quality Assurance

After experimenting with ChatGPT, an AI language model that has rapidly become indispensable for many individuals and even some businesses, I decided to explore its potential benefits for QA professionals. This experience provides an opportunity to optimize QA processes and quickly find solutions to questions using the capabilities of ChatGPT.

As a starting point, I engaged ChatGPT to uncover the benefits it could offer to software testers. In response, it highlighted several areas where it could provide its assistance:

• Test Scenario Generation
• Bug Reproduction and Reporting
• Test Data Generation
• Exploratory Testing
• Test Automation Assistance
• User Experience Testing
• Continuous Learning and Knowledge Sharing

Let’s touch on a few of the aforementioned areas and observe their outcomes.

Test Cases

For the Test Cases Generation I decided to choose an ecommerce website. I provided all the requirements in the prompt.

“Can you help me to create test cases with titles, preconditions, expected results for the new feature in an e-commerce website implementing a Free Shipping option for specific items?
Free Shipping badge should be displayed on Cart, Checkout and Confirmation pages. Not all items have Free Shipping.
Please create positive and negative test cases.”

ChatGPT generated the test cases with the following titles:

- Free Shipping Badge Displayed in Cart
- Free Shipping Badge Displayed in Checkout
- Free Shipping Badge Displayed in Confirmation Page
- No Free Shipping Badge in Cart
- No Free Shipping Badge in Checkout
- No Free Shipping Badge Displayed in Confirmation Page

One of the full Test Cases example:

Title: Free Shipping Badge Displayed in Cart

Preconditions:
The website has products with free shipping and products without free shipping.
A product with free shipping is added to the cart.

Steps:
1. Add a product with free shipping to the cart.
2. View the cart page.

Expected Result: The cart page should display a Free Shipping badge or indication next to the product that has free shipping.

Not bad!

Then, I realized that there was an important scenario missing when it comes to handling multiple items. To fill this gap, I decided to provide more information to the chat:

“Can you add test cases for multiple items in cart?”

ChatGPT returned a few more test cases with the following titles:

- Free Shipping Badge Displayed in Cart with Multiple Items
- No Free Shipping Badge in Cart with Multiple Items without Free Shipping
- Free Shipping Badge in Cart with Multiple Items, One with Free Shipping

It is worth mentioning that the ChatGPT returned well-structured titles, preconditions, and expected results.

Then, I became curious about any other scenarios that ChatGPT might have overlooked.

To explore this further, I added the following prompt:

“Can you see any other possible scenarios?”

ChatGPT generated some more test cases:

“Certainly! Here are a few additional scenarios and test cases to consider for the Free Shipping feature implementation:”

- Title: Free Shipping Eligibility based on Order Total
- Title: Free Shipping Promotion Applied
- Title: Free Shipping Re-evaluation after Item Removal

As you may have noticed, the last test cases were already based on assumptions. So make sure to always review the responses and align them with the project’s acceptance criteria to ensure the accuracy.

Limitation:
I can recognize certain limitations where extensive domain knowledge is necessary, which may restrict Chat GPT’s ability to generate highly accurate test cases at this stage.

Recommendations:
My recommendations after working on creating manual test cases with ChatGPT are as follows:

Firstly, it is crucial to construct clear and detailed requirements. The more specific you are with the requirements, the more accurate answers you will receive from ChatGPT.

Secondly, it is essential to review the responses provided by ChatGPT. Sometimes, it could make assumptions that may not necessarily be true and provide false positive or false negative results. Therefore, it is important to carefully assess and verify the responses.

Upon reviewing the responses, if needed, you can modify the question, provide additional information, request the regeneration of the response, and select the replies that you deem most suitable and relevant to the given prompt.

Test Data

Let’s see how ChatGPT can be helpful in generating test data.

My request:

“Test data for testing the mobile phone field which should accept only numeric characters. Mobile phones should have 10 digits. Please provide only negative test cases.”

Below is the outcome I received:

1. Mobile phone field contains alphabetic characters: “abcdefghij”
2. Mobile phone field contains special characters: “!@#$%^&*()”
3. Mobile phone field contains a mix of alphabetic and numeric characters: “abc1234567”
4. Mobile phone field contains fewer than 10 digits: “123456789”
5. Mobile phone field contains more than 10 digits: “12345678901”
6. Mobile phone field is empty: “”

I could use it.
Now what about generating test data for the excel sheets?

“Can you generate me random data for excel columns: first name, last name, phone, city, country — 20 rows of each?”

ChatGPT completed the task within a few seconds, undoubtedly faster than the time it would have taken me to enter the data manually or googling for test data sets.

Debugging

Next, I decided to check whether ChatGPT can identify the mistake I intentionally made in the SQL query.

“Can you check what’s wrong with my sql query:
SELECT customers_id, SUM(invoice.total) AS total_order_amount
FROM customers
JOIN invoice ON customers.customer_id = customers.customer_id
GROUP BY customers.customer_id
HAVING total_order_amount > 100
ORDER BY customers.customer_id
LIMIT 10;
”

And again BINGO!
It found the mistake, corrected it, and provided a very helpful explanation.

Looks like debugging will become much easier now :)

Automation Testing

I made a step forward and decided to check the possibilities of ChatGPT in automated scripts.

Can you create an automation script using Cypress in Javascript? I need a script for search functionality:
1. I go to the website “xyz.com”
2. I search for “mirror” and click enter key
Search box has an id “searchBox”.
I should be redirected to the Search page with h2 heading “mirror” results.

The response actually surprised me. It is a ready-to-use snippet of code that you can just copy. Make sure to test whether it’s actually working as intended :) At first glance, it appears to be very promising.

Then I asked to revise the script to align with my intent of utilizing it within the BDD approach.

“Can you remake it to BDD with Cucumber?”

And here is what I got:

Gherkin feature file

JavaScript file

Nicely written 2 files for the requested scenario.

It also provided suggestions on the necessary directories to be created and the dependencies to update. I am genuinely impressed with the results, which now pique my curiosity about how it will handle even more advanced test cases. It is something to explore in the future.

Conclusion:

ChatGPT excels in generating easy-mid complexity test cases, test data, and even automated scripts. I am sure it could serve anyone as a valuable virtual assistant, accelerating various tasks and freeing up time for other important projects. However, it’s important to comprehend both the strengths and limitations of ChatGPT to effectively harness its power. It can’t get your job done yet, however, by understanding its capabilities, you can certainly leverage ChatGPT to streamline your workflow and overcome certain challenges.