In this post, we will be talking about integration testing for Jet’s product catalog. Testing is a necessary part of software engineering because it allows us to:
- ensure that each feature developed operates as expected.
- save time taken by manual testing.
- increase design and implementation flexibility by having a specification expressed through test cases.
Also, it should be noted that it is impractical to expect testing to cover every permutation of business workflows in a complex system, but having good testing coverage does increase confidence in the feature release.
In what follows, we discuss details of integration testing in the broader testing landscape, we give a quick introduction to the Jet catalog system and accompanying terminology, then we talk about integration testing in the context of catalog system, and finally, we talk about an example of catalog API integration tests.
Integration Testing Overview
Integration testing is a way of testing software by grouping software components together. In a complex software system, there are numerous interconnections applicable to a particular feature, which makes writing integration tests difficult. This can be illustrated in Figure 1. In this workflow, we have test input data that is provided to a system component A, which then interacts with components B and C. The result of the output is then verified with the expected output.
In this context, there are several general components for integration testing:
- Developing features and writing a test case
Engineers start with writing code for features and their test cases. Once development is done, these features will be pushed to remote branches that would trigger the running of integration tests.
2. Creating a pristine integration testing environment
A dedicated rather than shared environment is needed since it gives testing good resource management, control and logging for the diagnosis and debugging purposes. Hence, each integration test run should start with a cleared environment.
3. Scheduling and running the tests
For each pull request of a new business feature, there should be a scheduled integration test build and run to validate that feature.
4. Reporting the result of integration tests
Finally, reporting the test result alerts team members of the status of feature development, and allows them to react quickly if there is an issue. This means that once all test cases have finished running, there should be a reporting procedure that would send a report with regards to the integration test results.
Catalog System Overview
Jet’s product catalog primarily focuses on the task of matching product offer sources and then representing them holistically so that other teams such as search, discoverability, and front-end can utilize this information. The system processes around 750,000 messages for product data updates and snapshot indexing per minute throughout the entire system on a typical day (peak traffic above 1 million messages per minute during holiday’s); it consists of a collection of micro-services, which interact through a message bus. Some services will perform a simple transformation on their inputs with relevant product and business information; others will enrich inputs with relevant updates to the product data snapshot (e.g. product data information for Nike shoes), or projections of the product data to databases for business analysis in response to these inputs.
Furthermore, several key entities in catalog system include offer source and SKU. An offer source is a way of representing raw product data (e.g. title, description etc) imported through merchants and is staged in the catalog initially. A SKU is a normalized view of all offer sources grouped by a unique identifier, such as UPC. Jet’s catalog system imports offer sources, matching them to corresponding SKUs, emitting an update event whenever the content of a SKU changes, either in response to merchant-initiated events, or manual updates. More specifically, an offer source will be merged to a SKU with appropriate pricing, image, and product data updates. In addition, there will be a SkuUpdate event for the SKU due to these updates.
Integration Testing in Catalog System
With this background information, we can write a simple test case to verify if multiple offer sources can create a SKU. The pseudo-code looks like:
Send pricing, product and image data for offer source "a"
Expect SkuUpdate event with offer source "a"The integration testing workflow would look something like:
Figure 2 shows how integration testing interacts with the catalog system. Firstly, it prepares and then sends pricing, offer source and image information through the message bus to relevant services in the catalog system. These services will then interact with SKU services that handle the updates to SKU from information originally provided by the integration test. Finally, the test case expects a SkuUpdate event from the catalog through the sku-updated-topic Kafka topic.
Domain Specific Language
Ideally, we would like the test case to be concise, extensible and reusable. F#’s computation workflow feature allows just that. It gives developers the flexibility to write code in a simplified and understandable format close to plain English. The logic in the above test case can be rewritten in the following DSL:
This DSL is a convenient way of writing test case since a statement that begins with a term such as Send and Get in the example above can be implemented once and reused multiple times in other test cases. It’s extensible because developers can extend the syntax of a statement to add further functionality to the test case. For instance, if a developer wants to send product data update for an offer source’s title field with a string, the syntax of statements that begins with Send can be extended as the following:
Integration Testing for Catalog APIs
Prior to adding integration tests, testing for catalog APIs needed to be completed manually with coordination between product owners and engineers. This manual testing requires the use of tools such as Postman to verify that API endpoints produce a desired output. This is time-consuming and quite inefficient, as each feature release means additional coordination between product and engineering team. Integration testing automates this process by systematically checking for inconsistencies in the API output. In addition, since most of our APIs serve as front gates to the system, it also checks that a component functions correctly through the API response. With the help of integration testing, several major features in the catalog system can be safely released with each deployment, including functionality related to product matching, taxonomy, and grouping.
In terms of the technical details, let’s say we have the following test case written in F# DSL:
At Jet’s catalog, one or more staged offer source can be merged to create a SKU with an approval message. Approved offer sources in SKU have “Merged” status. In the above example, we will be testing the API endpoint that sends an approval message to offer source: api/offersource/approve/{offer_source_id} where {offer_source_id} is ”a". With the background information on catalog system, we would expect a SKU created (hence a ”SkuUpdate" event for offer source ”a") and its offer source ”a" having ”Merged" status. Suppose that the offer source already exists with ”Staged" status. Then step one for the test case is:
The first two terms indicate that this is a test case for testing the offer source approval API. Terms after With provide the parameter for the test case; specifically, it specifies that the test case should send an HTTP approval message for the offer source "a" with payload ”test payload”.
DSL statements only specify “what” needs to be tested; there should also be a translation from “what” to “how” to test. The key information in the DSL statement, such as testing offer source approval API, offer source id, and payload information, can be encapsulated in F# variables in a library module, which can then be imported into the integration testing framework that runs the tests. Hence, an implementation for “how” to test a test case should be given in the framework. The implementation for testing the HTTP offer source approval API is:
The general flow of the code starts with sending the HTTP request for the ApiUrl constructed from the test case. Then it awaits for the OfferSourceStatusMerged event from the offer source to make sure that the HTTP request has just produced a response within the system that merges the offer source to a SKU. The awaitSpecificOfferSourceEvents function implements the functionality to await for a set of events of an offer source with a timeout.
The next two lines of code in the test case essentially check the test result by looking for a SkuUpdate event for the offer source "a" (this is because the offer source is merged to a SKU). It also checks the snapshot of the offer source to make sure that it has status "Merged".
Conclusion
Overall, integration testing is a good way to guard against bugs that might appear in a production system. It also saves development and debugging time since any issue can be fixed after the test case for that feature fails. Hopefully, the information presented in this blog post would be informative for testing your production system. Happy coding and testing!
If you like the challenges of building complex & reliable systems and are interested in solving complex problems, check out our job openings.
Acknowledgments
Designing and engineering a significant system is hardly an individual endeavor. The author would like to thank the Jet’s catalog team who have contributed to this work. For the post editing, the author is thankful for feedback from Ivan Glushchenko, Noah RobisonCox, James Novino, and Lev Gorodinski for advice on the structure and content of the post.