Automating Chatbot Testing with Python: From CSV to Excel with Detailed Analysis

Chatbots are becoming increasingly popular, providing users with instant responses, personalized experiences, and efficient service. However, as with any software, ensuring that your chatbot delivers accurate and relevant answers is crucial. Imagine you have a list of questions and expected answers — how do you automate the testing process? Even more, how do you generate an insightful report that not only tells you if the chatbot got it right but also explains why the answer might not be perfect?

In this article, I’ll walk you through how to set up an automated testing system for your chatbot using Python. By the end of this guide, you’ll know how to bulk test your chatbot with a list of questions, compare its responses with expected answers, and generate a comprehensive Excel report that includes a similarity score and a reason for each score. Ready? Let’s dive in!

Why Automate Chatbot Testing?

Before we get our hands dirty with code, let’s quickly talk about why you should automate chatbot testing:

1. Efficiency: Manually testing each chatbot response is tedious, especially as the number of test cases grows. Automation speeds this up.
2. Consistency: Automated tests run the same way every time, reducing the risk of human error.
3. Insightful Reporting: Automation allows you to generate detailed reports that can highlight not just whether the chatbot is performing well, but why certain responses might need attention.

Setting Up Your Python Environment

First, let’s set up our environment. We’ll need Python, along with some key libraries:

• pytest: For running the tests.
• requests: For sending HTTP requests to our chatbot.
• pandas: For handling our test cases and generating the Excel report.
• openpyxl: For writing Excel files.
• fuzzywuzzy: For comparing text similarity.

You can install these libraries using pip:

pip install pytest requests pandas openpyxl fuzzywuzzy

Organizing Your Project

Here’s the structure of the project:

/chatbot_test
|-- tests
|   |-- test_chatbot.py
|-- test_cases.csv
|-- config.py
|-- requirements.txt
|-- pytest.ini
|-- results
|   |-- test_results.xlsx

1. Configuration

In config.py, define the chatbot's endpoint:

# config.py
CHATBOT_ENDPOINT = "http://your-chatbot-endpoint/api"

2. Test Cases in CSV

Create a CSV file (test_cases.csv) with your test cases:

question,expected_answer
Hi,Hello! How can I help you today?
What is the weather today?,The weather today is sunny with a high of 25 degrees.
Tell me a joke.,Why did the chicken cross the road? To get to the other side!
Who is the president of the United States?,The current president of the United States is Joe Biden.

3. The Heart of Automation: test_chatbot.py

This is where the magic happens. Let’s break down the code step by step:

import pytest
import requests
import pandas as pd
from config import CHATBOT_ENDPOINT
from fuzzywuzzy import fuzz

# Send a message to the chatbot and get the response
def send_message_to_chatbot(message):
    response = requests.post(CHATBOT_ENDPOINT, json={"message": message})
    return response.json()

# Read test cases from a CSV file
def read_test_cases(file_path):
    return pd.read_csv(file_path)

# Write test results to an Excel file
def write_results_to_excel(results, file_path):
    df = pd.DataFrame(results)
    df.to_excel(file_path, index=False)

# Generate a reason based on similarity percentage
def generate_similarity_reason(similarity, expected, actual):
    if similarity == 100:
        return "Perfect match"
    elif similarity >= 80:
        return "High similarity"
    else:
        return "Low similarity: The expected answer and the actual answer are significantly different."

# Read test cases
test_cases = read_test_cases('test_cases.csv')

# Store test results
results = []

@pytest.mark.parametrize("index, test_case", test_cases.iterrows())
def test_chatbot_responses(index, test_case):
    question = test_case['question']
    expected_answer = test_case['expected_answer']
    response = send_message_to_chatbot(question)
    actual_answer = response.get("message", "")
    
    # Calculate similarity
    similarity = fuzz.ratio(expected_answer, actual_answer)
    
    # Generate reason
    reason = generate_similarity_reason(similarity, expected_answer, actual_answer)
    
    # Append the result
    results.append({
        "Question": question,
        "Expected Answer": expected_answer,
        "Actual Answer": actual_answer,
        "Similarity (%)": similarity,
        "Reason": reason
    })
    
    # Check if the similarity is acceptable
    assert similarity > 80, f'Expected similarity > 80%, but got {similarity}%'

# Write results to Excel after all tests
@pytest.hookimpl(tryfirst=True)
def pytest_sessionfinish(session, exitstatus):
    write_results_to_excel(results, 'results/test_results.xlsx')

if __name__ == "__main__":
    pytest.main()

Breaking Down the Code

• Sending the Question: The send_message_to_chatbot function sends your question to the chatbot and receives the response.
• Reading Test Cases: read_test_cases reads your CSV file into a DataFrame.
• Calculating Similarity: We use fuzzywuzzy to compare the chatbot’s response to the expected answer.
• Generating a Reason: Depending on how close the chatbot’s answer is to the expected answer, we generate a reason to explain the similarity score.
• Saving the Results: After all tests run, the results are saved to an Excel file, neatly organized.

4. Running the Tests

Simply run the tests with pytest:

pytest

After the tests finish, you’ll find a shiny new test_results.xlsx in the results/ directory. Here’s what it contains:

Conclusion

And there you have it! An automated testing setup that not only checks if your chatbot’s answers are correct but also provides an insightful analysis of why a particular answer did or did not match expectations.

Automating your chatbot testing process saves time, reduces human error, and provides detailed feedback that can help you continuously improve your bot. With Python, pandas, and a little bit of fuzzywuzzy magic, you've got a powerful toolset to ensure your chatbot is always ready to give the right answers.

Happy coding, and may your chatbot always be on point!