Mocking API request tests for Wagtail… with KanyeRest

I’ve been working on something(s) recently involving fetching data from an API and using it in a Wagtail site. I came to the point when I was struggling to test the API request without actually requesting the live API. Why? Well, sites can go down, take to long to respond or generally misbehave and really, my API request should be able to handle this gracefully. More specifically, it should expect it.

I actually love testing. So much so, I would be happy just writing tests all day but I find it hard just to figure out how to test specific things. This can cause me to skip writing some tests, hopefully those days are numbered. I wanted to share the solution because it’s blown my mind and opened a whole new area of testing to me.*

For this post, I’ve made a quick system for requesting quotes from an API and displaying them on the homepage of a Wagtail site. Here is the homepage models.py file:

from home.api_content import KanyeRest
from wagtail.core.models import Page


class HomePage(Page):
    def get_context(self, request, *args, **kwargs):
        context = super().get_context(request, *args, **kwargs)
        context["kanye_quote"] = KanyeRest().get_data()
        return context

Main thing to note is line 8, this is calling a method on my KanyeRest class which returns the quote data. So the homepage looks like this:

Homepage example showing a quote… I know, Kanye is a bit… creative.

I’m using Kanye.rest, it’s a free REST API for random Kanye West quotes…kudos to @ajzbc.

Example of data from api.kanye.rest

Here is the full KanyeRest class responsible for this madness, it’s what we will be testing:

import json
import logging

import requests
from django.conf import settings


logger = logging.getLogger(__name__)


class KanyeParse:
    def __init__(self, data):
        self.data = data
    
    # A very simple parser
    def get_parsed_data(self):
        parsed_data = json.loads(self.data)
        return parsed_data["quote"]


class KanyeRest:
    def __init__(self):
        self.data = None
        self.url = "https://api.kanye.rest/"

    def get_url(self):
        return self.url

    def fetch_data(self):
        try:
            response = requests.get(url=self.url)
            self.data = response.content
        except requests.exceptions.Timeout:
            logger.exception(f"Timeout occurred")
        except Exception:
            logger.exception(f"Error occurred")
        return self.data

    def get_data(self):
        """ If there is data, parse it, otherwise return an empty list """
        self.data = self.fetch_data()
        if self.data:
            self.data = KanyeParse(self.data).get_parsed_data()
            return self.data
        else:
            return []

In the real world you’d cache the response data, but for demo purposes this will do. Here is what I wanted to test on my project at the time and what we should test here:

• How does the code handle a Timeout?
• Does the data go through the parser as expected
• If there are Exceptions, will the homepage still render?

Whilst a test for checking the live API responds is useful, I’m just interested in what happens when the request fails. This is where the mock library comes in. Actually it’s where my buddy Rich came in with a bag full of mock (you may remember him from this post I credited him on).

unittest.mock is a library for testing in Python. It allows you to replace parts of your system under test with mock objects and make assertions about how they have been used.

That reference has probably been posted on Medium a thousand times but it’s clear and simple. How mock really helps with testing this stuff is it allows you to patch things and hi-jack what they are doing. In our case, we can mock the entire `requests.get` method, or just mock certain classes and methods.

The end result test_api.py file is here. But I’ll break it up…

Testing the API is up

Here, I’m setting up some values to test, seen as though I don’t want to use the live API because data changes with each request. Then a quick test_fetch to check the live API is responding.

class TestKanye(TestCase):
    def setUp(self):
        self.expected_api_data = {"quote": "All you have to be is yourself"}
        self.expected_parsed_data = "I want the world to be better! All I want is positive! All I want is dopeness!"
        self.default_data = [{"quote": "This is me, not Kayne"}]

    def test_fetch(self):
        url = KanyeRest().get_url()
        response = requests.get(url, timeout=5)
        self.assertEqual(response.status_code, 200)

Testing a fetch with mocked data

Using mock.patch can replace methods and classes with ones you define specifically for tests. Here I’m saying “Don’t use the fetch_data method from my KanyeRest() class, instead use mock_fetched_data”

# Data changes on each request so mock it
@mock.patch("home.api_content.KanyeRest.fetch_data", side_effect=mocked_fetch_data)
def test_fetch_with_example_data(self, mocked_fetch_data):
    data = KanyeRest().get_data()
    lf.assertEqual(data, self.expected_parsed_data)

This is a bit too general, but it’s worth showing as an example. Here, I’m testing the whole thing but overriding the data I get from fetch_data. Doing this allows me to check the end result data is what I expect (self.expected_data). The mocked method is defined in the test code outside the test class like this:

def mocked_fetch_data():
    data = json.dumps(
        {
            "quote": "I want the world to be better! All I want is positive! All I want is dopeness!"
        }
    )
    return data

Testing the default data is used with a Timeout

Here with mock.patch, I’m mocking the get request used by requests.get. Doing this allows me to define a ‘side_effect’, and that can be a Timeout exception.

@mock.patch("home.api_content.requests.get")
def test_data_if_timeout(self, mock_get):
        """ If a timeout is caught we should be getting the             default data"""
    mock_get.side_effect = Timeout
    data = KanyeRest().get_data()
    self.assertEqual(data, self.default_data)

But does the page render with a Timeout?

Yep, this test is similar to the test above, we just check the rendering of the page:

@mock.patch("home.api_content.requests.get")
def test_page_renders_with_timeout(self, mock_get):
    """ If there is a timeout for the request when the page loads,
    ensure the page still renders with the default data"""
    home_page = HomePage.objects.first()
    mock_get.side_effect = Timeout
    response = self.client.get("/")
    self.assertTemplateUsed(response, "home/home_page.html")
    self.assertEqual(response.render().status_code, 200)

This is the part where I say there is loads more to learn about Mock, test isolation and mocking too much can lead to a weird and almost useless test. For me, this was a massive jump in what I can actually write tests for, I highly recommend giving it a try, purely because requesting data from external API’s is very common.

*There are plenty of other posts on Mock and this wont be the last, they are definitely worth a read:

What the mock? - A cheatsheet for mocking in Python

Using Python Mock to Haunt Your Code

Python Mocking, You Are A Tricksy Beast