Automate to validate
A common distributive design has helped McDonald’s automate the validation process for delivery email receipts to make it faster and more accurate.
by Asitha Kaduwela, Software Engineer III
In Global Technology, we strive to deliver robust and maintainable solutions for our products. As we deliver these solutions, it’s important to ensure they’re properly tested as they move through various test environments and are eventually released into production.
In lower environments, we use the OpenTest framework to conduct automated validation of our services as we deploy changes. The intent is for the validation to do the heavy lifting. Changes get deployed, the scripts run, and within minutes we should have test results we can work with. So, whenever we find ourselves manually verifying the results of a code change, we ask: Can we automate this?
Emails
Recently, we asked this question about our delivery email receipts validation process. During the delivery lifecycle, there are various stages where a receipt is emailed to the customer.
For example, when your delivery order is created, you’ll receive an email containing information, such as the order’s items, applicable taxes, and more. There’s quite a bit of information in these emails, and manually verifying them has proven to be a time-consuming and daunting task. Not only that, but manual verification can also lead to uncaught human errors.
To illustrate our manual verification process, let’s look at our infrastructure for emails.
Previous setup:
This shows a simplified version of our email flow. Various email notification producers can publish messages to an SNS topic. Those messages are then forwarded to an SQS queue, which is subscribed to that topic. After that, email publication services will consume from the queue, and an email will get sent to the customer’s inbox. In our development environment, these emails arrive at a test inbox where we can open and visually validate them.
The challenge
The challenge here was that we wanted to automate a part of this validation process while conserving the original flow of generating the email itself. In other words, we needed a way to distribute a target payload in this flow instead of intercepting it, which would break the flow. Specifically, we focused on validating the SNS email payload that’s published from the email notification producers, which are the services our team owns. Ultimately, we found a solution that satisfied both of our requirements: the SNS/SQS fanout design.
New setup:
In the flow above, we added a new Test SQS queue that’s subscribed to the same SNS topic, which is used for the email generation flow. This results in each SNS message getting “fanned out” to both queues. Our OpenTest scripts are then able to consume from the test queue and validate the payloads.
Rapid feedback
With this setup, we’re able to get feedback on email changes within moments of deploying. This quick feedback is useful in lower environments where deployments can happen often. Any negative feedback from testing can give us clarity into what went wrong, so we can fix any issues before deploying to higher environments. The fanout enables us to conduct automated validation on our email payloads, while preserving the original flow of email generation. In addition, this design was attained in only two steps:
1. Create a new SQS queue that’s subscribed to the existing SNS topic
2. Consume and validate payloads in existing OpenTest scripts
This common distributive design helped enable us to find a solution that required minimal infrastructure changes. Not only has it proven to be useful here, but it’s also been used in other products as well and is a strong driver for automating processes within Global Technology.
