Best Practices for Unit Testing Emails

Have you ever wished your unit tests and staging environments fully tested your emails too?

Nerdy topic, but something we strive to solve at sendwithus.

The increasing dependence on cloud services necessitates a review of the software lifecycle and an analysis of how cloud services should be tested. Integrating a cloud service isn’t as simple as dropping in an API, especially when it impacts customer-facing aspects such as email. To properly investigate and add cloud services, it’s important to review how they will interact with your software testing — we like to break this down to local development, unit tests, QA or staging and environments, and finally your production environment.

Testing Email Locally

Testing software begins on your PC, while you’re writing code and working on that swanky new feature. Two key aspects of local testing are: the frequency at which you run such tests (crazy high) and the speed at which you need to perform them (super fast) — both of these are expressly affected if you’re relying on a third party API. An additional concern with a third party API is that there may be costs associated with every API call — in the case of email, almost every cloud SMTP provider (like SendGrid or SparkPost) will charge you per every email send which can rapidly add up the more engineers you have. Tempted to simply avoid these problems by ceasing to conduct email sends from developers’ local computers? (Bad form!)

It’s highly desirable for your development environment to be as close as possible to your production systems, and developers can’t fully ensure that their code is working if the “send email” system does nothing.

A better solution can be found in the open source project MailCatcher (https://mailcatcher.me/). MailCatcher runs a simple SMTP server locally and provides a browser-based interface to view all email sent to it.

However, there are still drawbacks to using MailCatcher: 1) you’re limited in your ability to use a cloud based template solution because MailCatcher only works as an SMTP server, 2) it can start to run slowly if you’re running your tests for a long time and testing many emails, 3) it’s a pain to have to run a separate service alongside your development environment, with heavy dependencies (Ruby).

A shining alternative is to use the sendwithus Test API Key, designed with this exact use case in mind. The beauty of using a Test API Key is that you don’t have to change any of the logic for your local development environment, just swap the API key in use. The primary benefit of the Test API Key is that it will (very quickly) verify that you sent a valid API request and immediately respond — without performing any other actions. It’s great for when you’re debugging a complex user flow because you can avoid the trouble of sending an email each time you test it, while enjoying verification that the email would be sent correctly.

Unit Tests and Continuous Integration

Once you move into Unit Tests and Continuous Integration (CI), the testing requirements around email change quite a bit: the main requirement of both Unit Tests and CI is that they can be run without any developer (or human) interaction. While it’s common to require Unit Tests and CI to send real emails, there are some regular pitfalls with this approach:

First, you run the risk of sending emails to people you didn’t mean to (at worst, real customers). Setting strong controls on your test environment to ensure real customer data is never used can help relieve this risk, though it’s not 100% fool-proof. Second, with most cloud based SMTP services, these test sends are unfortunately going to confuse your email analytics with all the real emails your product sends. Third, CI in particular is very powerful when your tests can go “full circle” — the example with email would be having an inbound SMTP server actually receive the email and then check that it matches some requirements specified in your test. Most modern cloud SMTP services provide an “inbound email API” that can specifically help with this problem.

The sendwithus approach is to provide a special kind of test key, called the Override Key. Override keys are great because they solve the first two problems of unit testing email. The “override” comes from a hard coded “destination email address” that you set up when you create the Override Key in the sendwithus dashboard. Using this, no matter “who” you try to send email to using the sendwithus API, email will be redirected to the corresponding destination email address. This removes all the risk of what data your unit tests use, as well as giving your development team a single inbox to view test email data. It’s quite common for a company to create a “unittests@company.com” catch-all for use with this API key. Of course individual developers can add their own override API keys if they need to.

Finally, analytics are muted completely for any emails sent via the Override API keys so you’re not polluting your production engagement data. Full logging is available for each send however, if you filter for “test” sends.

Staging and Quality Assurance (QA) Environments

Though it may often be the case that, once Unit Tests or CI have passed, code is deployed into production, this process is not uniform across all organizations. Most enterprise companies require a staging environment for user testing, quality assurance, or just plain old “human testing”. Staging environments encounter many of the same problems as Unit Tests and CI, with the added complexity of human testing errors. Similarly to unit testing, the sendwithus Override Keys can prove extremely useful here — human testers can run through the complete user creation and signup processes, then have all related email delivered to a common inbox. This entire process requires no code or control flow changes, just a different API key for the staging environment.

Finally, there are some scenarios for QA where emails will need to go to different recipients, however you would still want your testing analytics to remain separate from production. For this use case sendwithus has developed the Test-Send Key, which does exactly what it sounds like! Using this API key, emails will be delivered to the original intended recipient but no analytics will be tracked. Full logging will still be available in the sendwithus dashboard, making it easy for developers to debug.

To recap: with sendwithus you can create and manage as many API keys as you need, each API key can be given a unique name (specific to it’s use case and environment) then saved in your account once it has been set up. The 4 main keys are as follows:

Production Key

You guessed it: production keys will always send real email to real customers! Emails sent using production keys will show up in email logs and full scale analytics will be run. These keys should only be used when accessing the API in production applications.

Test Key

Test keys are perfect for setting up simple integration tests, where the emails themselves aren’t the primary focus. This type of key was designed to not send real emails and is intended for testing API calls, making this key the most useful for the initial stages of testing and development. This key is used primarily to make sure that the API calls you have setup to trigger an email from your application is working; so typically when you’re setting up transactional emails. For example, let’s say a user signs up to your website, you want to make sure that your application is making the correct Sendwithus API call to trigger an email, and that the API is returning with a successful response.

Meaning, you’re really only focused on making sure everything is valid for the API call. You’ll want to make sure your app is sending the correct data for the API call like the recipient’s email address, and of course, any relevant data like first name, to personalize the welcome email. Since you (a developer) will be doing this type of test locally on your PC a zillion times per day, to remain space/cost effective you wouldn’t want actual emails to be sent to an actual inbox, which is why you would use the test key (a lot).

Override API Key

At sendwithus we use override API keys internally for development, and we love them so much we wanted to offer them to everyone. These API keys let you direct all email sends to a single email address and are very useful in testing actual email sends. This key will allow you the freedom to test how various templates will render with the data you are passing through on the API call. Basically, they’re great for general testing and staging environments, especially when you want to send test emails whilst avoiding the potential to accidentally spam your customers.

Test-Send Key

Test-Send Keys are primarily utilized when you’re just “10% away from the finish line” and want to test everything: from system testing to extensive email rendering and functionality. These emails will be delivered to the specified recipients, so use this key with caution. This key will allow your QA team to send emails to other members of your team to get feedback on the emails. QA stages vary widely in design, so the QA round depends on what it looks like for your company. Generally, in this stage you can test to be sure the email renders as expected, the desired content is there, the template logic makes sense, and the email address was delivered to the user. These keys behave similarly to production API keys, except the analytics won’t be tracked. You can find additional information on these sends that use this key, on the test logs page. This will detail what template was delivered, what version was sent, which ESP was used, and more.

#EmailLegends

As more ownership of email is passed on to marketers who work heavily with templates, it is wise for developers to work in concert to troubleshoot and verify proper functionality of these emails — we’ve designed a method for developers to be able to build and deploy code with complete confidence that users will get the email that they are scheduled to get, and that they will render and click-through the way they ought to.

To learn more about sendwithus and how our platform can help you manage an efficient email testing environment, please drop us a note at sales@sendwithus.com … or if you’re the shy type, lurk our website at www.sendwithus.com