Why Mocking Data is a Bad Practice for Testing

7 min readJul 4, 2023

Mocking data is a common technique for testing software that depends on external sources of data, such as databases, web services, or APIs. Mocking data means creating fake or simulated data that mimics the real data, but without actually interacting with the external source. Mocking data can be useful for some scenarios, such as:

Isolating the unit under test from other components or dependencies that are not relevant for the test.
Speeding up the test execution by avoiding network calls or database queries that can be slow or unreliable.
Controlling the test input and output by creating predictable and consistent data that can be easily verified.

However, mocking data also has some serious drawbacks and risks, such as:

Introducing errors or inconsistencies between the mock data and the real data, which can lead to false positives or false negatives in the test results.
Reducing the coverage and confidence of the test, by not testing the actual behavior and logic of the external source or the interaction with it.
Increasing the maintenance and complexity of the test, by requiring extra code and logic to create and manage the mock data.

In this blog post, I will argue that mocking data is a bad practice for testing software, and that you should avoid it as much as possible. I will also show you some examples of how to test software that depends on external data sources without mocking data, using Python and PyTest.

An Example of Mocking Data

Let’s say we have a function that calculates the average rating of a product based on the reviews from an online store. The function takes the product ID as an input and returns the average rating as an output. The function uses an API to fetch the reviews from the online store.

import requests

def get_average_rating(product_id):
    """
    Returns the average rating of a product based on the reviews from an online store.
    """
    url = f"https://online-store.com/api/reviews/{product_id}"
    response = requests.get(url)
    if response.status_code == 200:
        reviews = response.json()
        ratings = [review["rating"] for review in reviews]
        average_rating = sum(pratings) / len(ratings)
        return average_rating
    else:
        raise Exception(f"Failed to get reviews for product {product_id}")

To test this function, we might be tempted to mock the data returned by the API, using a library like unittest.mock or pytest-mock. For example, we could write a test like this:

import pytest
from pytest_mock import mocker
from average_rating import get_average_rating

def test_get_average_rating(mocker):
    # Arrange
    product_id = 123
    mock_reviews = [
        {"rating": 5, "comment": "Great product!"},
        {"rating": 4, "comment": "Good quality."},
        {"rating": 3, "comment": "Not bad."}
    ]
    expected_average_rating = 4.0
    mocker.patch("requests.get")
    requests.get.return_value.status_code = 200
    requests.get.return_value.json.return_value = mock_reviews

    # Act
    actual_average_rating = get_average_rating(product_id)

    # Assert
    assert actual_average_rating == expected_average_rating

This test creates a mock object for requests.get and sets its return value to mimic a successful response from the API with some fake reviews. Then it calls the function under test and asserts that it returns the expected average rating.

The Problems with Mocking Data

At first glance, this test might seem fine. It runs fast, it passes, and it covers the main logic of the function. However, there are several problems with this test that make it unreliable and ineffective.

The Test is Not Testing the Real Data

The first problem is that the test is not testing the real data that comes from the API, but only a simulated version of it. This means that there might be differences or discrepancies between the mock data and the real data that could affect the behavior or outcome of the function.

For example, what if the real API returns more fields than just rating and comment? What if some of those fields are required or used by the function? What if some of those fields have different names or types than expected? What if some of those fields have invalid or unexpected values?

All these scenarios could cause errors or bugs in the function that would not be detected by the test, because it only uses a simplified version of the data. The test would pass even if the function fails with real data.

The Test is Not Testing the Interaction with the External Source

The second problem is that the test is not testing the interaction with the external source, but only a mocked version of it. This means that there might be issues or failures in the communication or integration with the API that could affect the behavior or outcome of the function.

For example, what if the API is down or unavailable? What if the API is slow or unresponsive? What if the API returns an error or a different status code than expected? What if the API changes its format or structure without notice?

All these scenarios could cause errors or bugs in the function that would not be detected by the test, because it only uses a mocked version of the response. The test would pass even if the function fails with real data.

The Test is Hard to Maintain and Update

The third problem is that the test is hard to maintain and update, because it requires extra code and logic to create and manage the mock data. This means that there might be errors or inconsistencies in the test code itself that could affect the reliability or validity of the test.

For example, what if the mock data is outdated or incorrect? What if the mock data is incomplete or missing some cases? What if the mock data is duplicated or inconsistent across different tests? What if the mock data is hard to read or understand?

All these scenarios could cause errors or bugs in the test code that would affect the test results. The test would fail or pass for the wrong reasons.

How to Test Without Mocking Data

So, how can we test software that depends on external data sources without mocking data? There are several possible solutions, depending on the context and the nature of the data source. Here are some general guidelines and tips:

Use a real or live data source whenever possible. This is the best way to ensure that your test is testing the real data and the real interaction with the external source. However, this might not always be feasible or desirable, due to factors such as cost, availability, security, or privacy.
Use a local or in-memory data source whenever possible. This is a good alternative to using a real or live data source, as it allows you to create and control your own data without relying on an external source. However, this might require some extra setup and configuration, and it might not always be compatible or consistent with the external source.
Use a fake or stubbed data source whenever possible. This is a good alternative to using a mocked data source, as it allows you to simulate a real response from an external source without creating fake data. However, this might require some extra code and logic, and it might not always be accurate or realistic.

Let’s see how we can apply these solutions to our example of testing the function that calculates the average rating of a product based on the reviews from an online store.

Using a Real Data Source

One option is to use a real online store API that provides reviews for products. For example, we could use Amazon’s Product Advertising API, which allows us to search for products and get their ratings and reviews. To use this API, we would need to register for an account and get an access key and a secret key.

We would also need to modify our function slightly to use this API instead of the fake one:

import requests
from amazon.paapi import AmazonAPI

def get_average_rating(product_id):
    """Returns the average rating of a product based on the reviews from Amazon."""
    amazon = AmazonAPI(ACCESS_KEY, SECRET_KEY)
    product = amazon.get_product(product_id)
    reviews = product.reviews
    ratings = [review["rating"] for review in reviews]
    average_rating = sum(ratings) / len(ratings)
    return average_rating

Then we could write a test like this:

import pytest
from average_rating import get_average_rating

def test_get_average_rating():
    # Arrange
    product_id = "B07XJ8C8F5" # A random product from Amazon
    expected_average_rating = 4.6 # The actual average rating from Amazon

    # Act
    actual_average_rating = get_average_rating(product_id)

    # Assert
    assert actual_average_rating == expected_average_rating

This test uses a real product ID from Amazon and checks that the function returns the same average rating as shown on Amazon’s website.

The advantage of this test is that it uses real data and real interaction with an external source, which increases its coverage and confidence. The disadvantage is that it depends on an external source that might not be available or reliable, which decreases its speed and stability.

Using a Local Data Source

Another option is to use a local database that contains reviews for products. For example, we could use SQLite, which is a lightweight and self-contained database engine that can run in memory. To use SQLite, we would need to install it and create a database file with some sample data.

We would also need to modify our function slightly to use SQLite instead of the fake API:

import sqlite3

def get_average_rating(product_id):
    """Returns the average rating of a product based on the reviews from a local database."""
    conn = sqlite3.connect("reviews.db")
    cursor = conn.cursor()

    # Query the database for the reviews of the product
    query = "SELECT rating FROM reviews WHERE product_id = ?"
    cursor.execute(query, (product_id,))
    reviews = cursor.fetchall()

    # Calculate the average rating
    ratings = [review[0] for review in reviews]
    average_rating = sum(ratings) / len(ratings)

    # Close the connection
    conn.close()

    return average_rating