ChatGPT for Software Testers: my experiment

As someone with a scientific background, I decided to do my own experiment with ChatGPT.

The Hypothesis: ChatGPT can help me become a more efficient and effective Software Tester

The Method: Don’t read any article. Use only ChatGPT and my brain 🧠

First Prompt

Me: “How can you help Software Testers excel at their role?”

Why not use the tool itself to understand where to start? This got me plenty of topics to delve into!

I chose a few and used a similar approach to begin each one’s exploration:

Me: “What’s the best way to ask you to [do something]?”

This helped me understand how to refine the prompts to get to the best possible outcome, tailored to my needs.

Here goes what I learned!

Generate Test Cases

After asking ChatGPT for the best way to prompt it to generate test cases, these are my top 3 recommendations:

1. Clearly specify the functionality (this one seems obvious, right? but the more specific you are and the more context you provide, the more time you’ll save in the 3rd point — trust me)
2. Ask for specific types of tests (functional, positive/negative, corner cases)
3. Review and Refine (in the absence of details it will make assumptions that may not be right! In that case, you should re-prompt it for the corrections)

Now, following the tips and an actual example of a prompt (provided by the very own ChatGPT), this was what I wrote:

Me: “Please generate 10 functional test cases for a document upload feature on a mobile application. Consider both valid and invalid inputs, including edge cases such as unsupported documents and corrupted files. Focus on testing the navigation and error handling. Please provide the test steps, expected results, and any preconditions or setup needed for each test case.”

The titles of the Test Cases suggested by ChatGPT:

1. Valid document upload
2. Empty document upload
3. Invalid document format
4. Large document upload
5. Upload cancellation
6. Network disruption during upload
7. Duplicate document upload
8. Document preview display
9. Error handling — server unavailable
10. Error handling — file corruption detection

And the structure of one of them:

Test Case 3: Invalid Document Format

Preconditions: User is logged in to the mobile application.

Test Steps:
Navigate to the document upload screen.
Select a document file in an unsupported format (e.g., TXT, XLS).
Tap the “Upload” button.

Expected Result: An error message is displayed indicating that the selected document format is not supported.

Pretty good for the information I provided, right?

Test Data Generation

In this case my top 3 recommendations are different:

1. Be specific about structure, format and required fields
2. Include variations such as ranges and constraints
3. Specify the quantity of entries needed

Me: “Given the following JSON, generate another for a Portuguese person that is more than 18 years old.

{
“first_name”: “Jane”,
“last_name”: “Doe”,
“dob”: “1990–01–31”,
“address”: {
“building_number”: “100”,
“street”: “Main Street”,
“town”: “London”,
“postcode”: “SW4 6EH”,
“country”: “GBR”
}
}’
“

The JSON returned:

{
 "first_name": "João",
 "last_name": "Silva",
 "dob": "2002–05–16",
 "address": {
   "building_number": "100",
   "street": "Rua Principal",
   "town": "Lisboa",
   "postcode": "1000–001",
   "country": "PRT"
 }
}

Perfect! So, if we need test data that represents many people from different locations, ChatGPT can quickly and accurately generate it for us!

Scripting

Again, another set of top 3 recommendations:

1. Clearly state the purpose and target system
2. Specify the input and expected output/desired outcome
3. Include any preferred tools or frameworks

Thinking about the Document Upload feature, I thought that it would be handy to have a script to generate test files with different sizes.

Me: “Please write a bash script to run on macOS that generates 5 sample PDF files with different sizes (from 100kb to 10mb)”

It not only generated the script but it also provided run instructions and details about the PDF files (white background, black letters, etc). I was really impressed! But the ultimate test was to see if it actually worked.

It didn’t work. First attempt to run the script, it throws an error. But do you know the best part? You can go back and ask for a fix.

Second attempt, it generated more than the 5 files initially requested. No problem, ask it for a fix again.

It worked on the third iteration. So, lesson learned: don’t rely blindly on what you get.

Test Automation

In the realm of test automation, I wanted to see if ChatGPT could understand public API documentation and generate automated tests accordingly. The recommended principles for this prompt align with the scripting prompt.

Me: “Please write an automated test using Ruby and RSpec to cover the User Group Creation as per the following API documentation: https://api.slack.com/methods/usergroups.create

Make sure to also include the verification of the response schema”

WOW! Now I was really impressed. The endpoint was right. The request body had the mandatory fields and two of the optional fields as well. Assertions were included for the status code and the schema, as requested. It assumed the response schema was defined on a separate file, ‘user_group_create_response_schema.json’, and gave me instructions to create the file. I asked ChatGPT to create it for me. At last, only thing I needed to do was to replace the API Token: everything else was ready.

What about testing one of the expected errors? I asked it to also cover the error ‘name_already_exists’. I realised the test assumed I already had at least one user group created, whose name I should replace in the test. So I asked it to create the existing user group as a setup step. This is what was added:

  before(:all) do
    # Prepare request payload to create the existing user group
    payload = {
      name: 'existing_user_group',
      channels: 'C12345,C67890',
      validate: true,
      token: token
    }

    # Send API request to create the user group
    HTTParty.post("#{base_url}#{endpoint}", body: payload)
  end

Now, keeping the context, I tried to go a step further.

Me: “Please write a K6 load test for the previous endpoint”

Although I did not provide any details, ChatGPT generated a load test script with specific stage definitions and performance thresholds. Those can be customised afterwards.

Not bad! Another task completed with success.

Continuous Testing and CI/CD

Already knowing ChatGPT impressive skills on generating, explaining, troubleshooting and fixing code, I decided to do something different here.

Part of the valuable work as Test Engineers is to make processes better. It can be really time consuming and requires great knowledge and vision. Could I chat for a while with ChatGPT and get some actual improvement suggestion I could implement?

Me: Tell me questions I can ask you to improve my Continuous Testing and CI/CD solutions
ChatGPT: (…) What are the recommended approaches for handling flaky tests in a CI/CD pipeline? (…)
Me: What are the recommended approaches for handling flaky tests in a CI/CD pipeline?
ChatGPT: (…) Retry Mechanism: Implement a retry mechanism for flaky tests (…)
Me; How can I rerun flaky tests automatically in a gitlab pipeline?
ChatGPT: (…)
test:
script:
— run_flaky_tests.sh
retry: 3
(…)

ChatGPT answers were shortened, but you get the point! See how you can go from really generic to really specific in a few seconds. You can use this approach for pretty much anything that you would like to have improvement suggestions on. You can also start more specific, of course, if you already have an idea.

Test Documentation and Reporting

ChatGPT told me it is great at generating documentation and providing nice summaries from a lot of content (such as test results). I decided to believe in it this time. Only thing I asked was to generate a Decision Log and, once again, I was impressed. How many times have you felt that you know very well what you want to say but spend a lot of time just constructing the sentences properly? Here’s your assistance!

Conclusions

Hypothesis confirmed to be true! But the magic comes from your prompts, interactions and reviews! Use it wisely! 🪄

ℹ️ Keep in mind that ChatGPT training data only goes up until September 2021.
ℹ️ Be careful of the information you are sharing with the tool: do not include any personal and sensitive data.