Thorough and reliable unit-testing of emails in django

How to properly write tests for emails

Warm up

I am pretty sure we can all agree on the fact that unit-testing is a great thing. Building services, encapsulating logic, writing tests for them. It works, it makes code more reliable and everybody is happy. So far, so good.

The moment you want to test code which leaves the cozy area of your code interacting with itself, like having a scheduled task or talking to an external API, things get a little more complicated. There are lots of techniques like integration tests to tackle these hardships. Still, it’s harder to write a regular unit-test and therefore you need to give it a deeper thought.

Photo by Carol Jeng on Unsplash

In medias res

One of those topics that at least feels “external” is emailing. Most web software you build nowadays will send your users content in form of an email. Maybe a password, maybe an update about something.

Without a best-practice at hand and no central course of action, every developer started testing our email services in different ways.

There are basically two approaches. You test what you put into the django mailing factory and trust its outcome. On the other hand, you can let your service or function generate the email and check the output.

If you are within a django TestCase all email traffic will be blocked and the generated mails, which would have been sent, are stored in a variable:

from django.core import mail
mail.outbox

So which approach is the better one? Personally, I prefer the second one because you can just trigger the email creation in your test, not caring about splitting up the logic or returning things you would not return otherwise just to be able to test it.

The downside of this practice is that the email object feels a little interlaced. For example, to get to the HTML content, you have to access it like this:

mail.outbox[0].alternatives[0][0]

Preliminary result

As we have seen so far, we have two challenges:

1. We do not have a clear pattern on how to unit-test emails
2. Testing the intestines of an email object might feel a little bit bulky

To address both these topics we implemented a testing class which wraps up most the use-cases and additionally provides you with django-ORM-like methods to properly unit-test.

Photo by Jeff Sheldon on Unsplash

Setting up the solution

Our team created a generic test class, included in our “django-pony-express” package, which covers most of the things you might want to test.

You can install the latest version of our package via pip (or pipenv ) like this:

pip install django-pony-express

In the next step you need to initialise the class in your testing class. Because the class itself behaves more or less like a singleton you can do this in the setupTestData(). It will be executed only once per test class and — compared to setUp() — not for every test.

from django_pony_express.services.tests import EmailTestService
from django.test import TestCase
class MyTestClass(TestCase):

    @classmethod
    def setUpTestData(cls):
        cls.email_test_service = EmailTestService()

Now you can use the service in all of your tests. If you know that you want to test emails in several classes, it makes sense to create a base test class which the other classes inherit from.

Awesome feature: Querying for emails

To provide a djangoesque look-and-feel, our test service can query the mail outbox for certain criteria.

The easiest thing to do is get all emails:

list_of_emails = self.email_test_service.all()

Maybe you want a specific one? You can filter for a given subject or a recipient, meaning the to, cc and bcc attributes:

# Get all mails with given subject
my_email = self.email_test_service.filter(subject='Nigerian prince')
# Get all mails with given to-recipient
my_email = self.email_test_service.filter(to='foo@bar.com')

Of course, you can connect filters as well:

my_email = self.email_test_service.filter(
    subject='Nigerian prince', 
    to='spambot@bar.com',
    cc='copy@bar.com',
    bcc='blind.copy@bar.com'
)

Once you have a queryset of emails — meaning the result of one of the queries above — you can work with it.

You can count the results, especially good for assertions:

# Count the number of results
quantity = self.email_test_service.all().count()

Do you need the first or last element in the list?

# Get first item in list
first_mail = self.email_test_service.filter(subject='Nigerian prince').first()
# Get last item in list
last_mail = self.email_test_service.filter(subject='Nigerian prince').last()

It is very common that you expect one specific email, and you want to know if your mail queryset contains exactly this element. So there is a helper function for this:

# Returns `True` if the queryset contains exactly one item
self.email_test_service.filter(subject='Nigerian prince').one()

Photo by Nathan Dumlao on Unsplash

Awesome feature: Assertion helpers

Usually, there are only a couple of ways you can blackbox-test emails. You want to ensure the subject, recipient or content functions the way you expect it to be.

That’s why the class provides some shortcuts which wrap the assertion.

1. You want to check for exactly one result?

self.email_test_service.filter(to='foo@bar.com').assert_one()

2. Do you want to check the quantity of found emails?

expected_number_of_emails = 1
self.email_test_service.filter(to='foo@bar.com') \ .assert_quantity(expected_number_of_emails)

3. Do you want to check the subject?

self.email_test_service.filter(to='foo@bar.com') \  
    .assert_subject('Reset password')

Finally, we have a look at the content of the email. As you probably know, an email object consists of two parts. A plain-text and an HTML part. To avoid checking two times, the wrapper assert_body_contains() checks in both parts and fails if the given string is missing in one of them.

4. You want to check if a certain string is included in the body?

self.email_test_service.filter(to='foo@bar.com') \
    .assert_body_contains('inheritance')

5. Sometimes you want to check if something is NOT part of the body…

self.email_test_service.filter(to='foo@bar.com') \
    .assert_body_contains_not('scam')

To make your life a little easier and happier, each of these methods takes an optional parameter msg which is passed to the assertion and will be shown if it goes sideways. Here is an example:

self.email_test_service.filter(to='foo@bar.com') \
    .assert_body_contains('inheritance', msg='Missing words!')

Best practices

After using the class to refactor the email-unit-tests of several projects, here is our approach we use in most cases:

1. Execute the service or function which generates the emails
2. Check if an email exists which has the correct subject and goes to the expected recipient
3. Check for some specific content in the body so you are sure the correct template is used (and text and HTML parts are consistent)
4. If you have special cases and the mail has in one case a variation a, write a test for both cases and check in the first if a is there and in the latter if a is NOT there.

What is your opinion on this topic?

Hope it works out for you as nicely as it did for us. Furthermore, did you make similar experiences? Did you go with a different approach? I’d be happy to get some feedback on this topic.