E2E Testing for Modern Cloud-Based Applications
Building an Order Management System for a Large eRetailer
The process of testing and validating the proper functioning of large systems that integrate multiple third-party components is not always a straightforward task. With modern solutions generally incorporating multiple interconnected processes and systems that work in concert, corner cases are often found that can have unforeseen consequences if not previously identified and addressed using comprehensive testing strategies.
The TrackIt team was tasked by a large e-retailer to remake its entire OMS (Order Management System) and build a solution that integrates multiple third-party APIs (Shopify, Accertify, Avalara, etc.) used by the company. The project was part of a larger retail strategy that involved moving the customer from an old-school monolithic legacy systems approach to a more modern, scalable, and cost-efficient cloud approach leveraging state-of-the-art AWS tools and services.
This article describes the various challenges and scenarios faced by the TrackIt team and provides insight into the approaches explored in order to test and validate the proper functioning of the OMS.
E2E Testing: What It Is & Why It’s Important
E2E (End-to-End Testing) testing refers to a software testing methodology that tests an application or a system’s workflow from the beginning to the end. The E2E testing approach aims to replicate real user scenarios so that the system can be validated for integration and data integrity.
E2E testing determines whether the various dependencies of an application are working accurately and also serves as an effective means to validate that information is being correctly communicated between the multiple components of the system.
How the E2E Tests Are Executed
The TrackIt team leveraged AWS CodeBuild, AWS’s fully-managed build service, to conduct the E2E tests. The team has an AWS CodeCommit repository where it pushes the code and every time a PR (Pull Request) is created, a code pipeline is triggered. AWS CodeBuild deploys the entire OMS system (which includes the deployment of a system of Serverless APIs, AWS Lambda functions, AWS Step Functions, and Amazon SQS queues) in a unique environment on the AWS Cloud that simulates the production environment.
Each build returns a message to the team to notify them whether the build failed or not. All the resources deployed to run the tests are automatically deleted at the end of each build to avoid unnecessary costs. This approach enables the team to execute multiple E2E tests simultaneously, yet cost-efficiently.
Scenario & Challenges faced by the TrackIt team
Approach #1: Fake Payloads
The TrackIt team’s initial approach was to build E2E tests with fake payloads (data sent by Shopify to the OMS). The team created payloads, pushed them to the system, and proceeded to check if the SQS messages were being sent (an indication that the workflow was functioning without errors).
Drawbacks of the Approach
This approach proved to be overly time-consuming and inefficient because multiple bugs would not be detected and the team kept running into the same problems over and over again. These regressions slowed down the entire process.
Approach #2: Framework with Payloads (“Golden Testing”)
Having acknowledged the inefficiency of the first approach, the TrackIt team came up with a second approach called “Golden Testing”.
How “Golden Testing” Works & Its Benefits
There are 50 “golden orders”, or scenarios, that represent the edge cases of the application. Instead of writing the logic for each of these orders, this second approach only requires the payloads (data sent by Shopify and other services) and a description of the orders (all the steps the order data needs to go through and all the information it needs to push the order out).
The approach requires the UAT (User-Acceptance Testing) team to put together a spreadsheet that contains all the bugs. To find bugs, the UAT team executes manual tests. They place orders on Shopify, the order management system receives these orders, and based on the responses that are provided, the UAT team fills a spreadsheet containing details of all the bugs/errors found in the system as a result of these manual tests. The TrackIt team then uses the information provided in the spreadsheet to identify and isolate errors within the OMS.
Despite still being time-consuming, this second approach provided the team with a deterministic approach to prevent regressions.
The “Golden Testing” Test Script
E2E Testing vs. Unit Testing
Working on the OMS gave the TrackIt team an opportunity to assess the importance of considering different options for testing to identify the most comprehensive and efficient approach. The team recognized that despite being easy to write, E2E tests also introduce a fair degree of complexity to the process of testing intricate systems containing multiple integrated third-party components.
- Benefit: Easier to Execute
E2E tests are easy to write when an entire system needs to be tested as a whole. Using Unit tests to comprehensively test a system would be a much lengthier and less efficient process.
- Drawback: Slow
One of the fundamental drawbacks of running E2E tests is that they take a long time to be completed. Using E2E tests for bugs that could have been resolved with Unit tests can slow things down. This proves to be an issue when one compares the time required to run the 50 E2E tests needed for this project (which takes 5–10 mins in this example with the 2nd “Golden Testing” approach) to the time required to execute all the Unit tests (10 seconds).
- Drawback: Difficulty of Identifying Bugs
Contrary to Unit testing where each component of the system can be tested in an isolated manner, E2E testing does not provide testers with the ability to immediately identify where a specific error has occurred.
A Combination of E2E Testing and Unit Testing
Having acknowledged the limitations in efficiency arising from choosing to solely execute E2E tests, the TrackIt team’s final approach was to execute more Unit tests while continuing to leverage “Golden Testing”. This addition of Unit tests provided the TrackIt team with the necessary flexibility to isolate and test individual components almost instantly, whereas continuing to use “Golden Testing” enabled the prevention of regressions.
TrackIt is an Amazon Web Services Advanced Consulting Partner specializing in cloud management, consulting, and software development solutions based in Venice, CA.
TrackIt specializes in Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization with specialized expertise in Media & Entertainment workflows, High-Performance Computing environments, and data storage.
TrackIt’s forté is cutting-edge software design with deep expertise in containerization, serverless architectures, and innovative pipeline development. The TrackIt team can help you architect, design, build, and deploy customized solutions tailored to your exact requirements.
In addition to providing cloud management, consulting, and modern software development services, TrackIt also provides an open-source AWS cost management tool that allows users to optimize their costs and resources on AWS.