Midway: Walmart’s mocking journey…
Most of the applications rely on one or more back-end services. For successful test execution and fast development, all the dependent backend services should be reliable 100% of the time. However, to keep services up with consistent data for development and testing is very challenging, which makes testing/development against non-prod services inefficient and time consuming.
At Walmart, hundreds of engineering teams faced this same challenge as dependency tree at Walmart scale is enormous and may easily involve over 100+ teams for one end-to-end functional flow, resulting in slow development, test execution and impacting test reliability — especially, when there are false-negatives in test results for applications caused due to dependent services intermittent time-outs, unavailability, etc., as it becomes very challenging and time consuming to debug and evaluate root cause simply because these issues are not easily reproducible.
Way Forward…
To keep up the velocity of test automation across Walmart it was important to build confidence in test results by reducing false-negative which made it very evident that we needed a mocking solution that can address Walmart needs by eliminating testing and development dependencies on back-end services and can operate at Walmart scale.
Back in 2015, during our brain storm sessions and initial evaluation of available tools, it became very clear to us that the tool’s DNA should have the following:
- Easy Set-up: First time user should be able to set-up a mock server and should be able to mock at least couple of APIs under 10 minutes — this will fuel faster adoption and teams don’t have to feel burden of learning a new complex tool for their day to day work.
- Library not a server: The solution should provide a way to mock REST APIs without being centrally hosted so that it can be started and stopped for the purpose of development and/or testing locally or in CI/CD as needed by individuals or teams to eliminate any network dependencies and could work anywhere.
- Massive Parallelism: Able to run test cases in parallel (Walmart runs over 300,000+ test cases every day for front-end alone) to save time and resources.
- Decoupling: No coupling with user code — Development or Test code should be independent of mock implementation so that integrating mock server does not require overhaul of existing set-up or process.
- Multi-Platform: The solution should be able to support Web, mWeb, Native and Services development and testing.
- Documentation: Getting a tool adopted by hundreds of teams means a lot of support hours and therefore the documentation should be top-notch so that we spend more time in feature development than support.
Solution
Named after USS-Midway, Test Armada’s open source mocking fleet Midway provides powerful set of tools to address above mentioned challenges at scale. Some of the features supported are:
- UI Interface: Mock service UI for manual testing/debugging.
- Drop-And-Respond: Respond with a JSON file based on the route path automatically by dropping response file in folder.
- Variants: Switch to different data set for REST API.
- Test Reuse: Execute same test cases against mock or live service.
- Swagger integration: Automatic mock creator for web-services with swagger definition.
- Common Utilities: Common utility methods are provided as part of the mocking service which allows quicker test development.
- Mock from container: Mocked data response from specific directory irrespective of Rest APIs
- Parallel Sessions: Support for single instance mock server for parallel processes.
- Share Mock data: Allows fetching of mocked data and routes from multiple Git repositories — allowing teams to share their mocked responses.
- Dynamic Transposition of Mock Data (JSON): Ability to modify response on the fly.
- Support for all file types: Auto evaluation of response file extension and mime type
- Platform independent mocks: Mock any service irrespective of the language it is written in.
- Server states: Ability to mock server state.
- Manual tests: Ability to run tests manually against mock service
- Service Faults: Team can simulate service faults deterministically.
Three years before when we started there were not many open source mocking tools that could suffice our requirements, however, since then mocking for functional test automation and development (not unit test cases) has gained lot of momentum. Some other noteworthy open source mocking tools are Polly.JS, Sinon.JS, Mimic, etc. each solving one or more distinct use case.
Case Studies
Point of sale manual/automated test cases and software development.
Challenges:
- External service down impact for development was 10 hrs/wk
- External service down impact for testing was 12 hrs/wk
- External service change impact incidents were 2 per week
- Test flakiness was at 20%
- Negative test scenarios were zero.
Midway Impact:
- External service down impact for development reduced to 0 hrs/wk
- External service down impact for testing reduced to 0 hrs/wk
- External service change impact reduced to 0 per week
- Test flakiness reduced to 5% (expected to reduce to 0% once all services are mocked)
- During first week of POC, 10+ negative test scenarios were added.
Automated test cycles for Walmart.com front-end.
Challenges:
- Complete test cycle took too long, and in some cases days to finish.
- Due to longer test cycles, the PR and Master verify were done on limited browsers.
- Only 32 test cases could be run in parallel due to CPU limitation caused by starting of multiple app and mock server, resulting in requiring more hardware for test execution — took 25+ minutes with 4 servers each to execute 400 test cases.
Midway Impact:
- Test execution time was reduced by 40% to 60% for individual teams.
- PR and Master verify are run on all top tier browsers due to gain in test execution speed.
- Could run 120+ test cases in parallel reducing the hardware cost by almost 10 times — took under 10 minutes with 1 server to execute 400 test cases.
At Walmart, all front-end test automation teams uses Midway for mocking back-end services for their test automation. Using Midway as a mocking solution enabled faster test execution, reduced false negatives and helped optimal use of resources thus reducing the cost and increasing overall efficiency. In less than a year, Midway had already mocked over 185+ million REST APIs.
Stay tuned for future blogs for in-depth architecture and features explanation. In the meantime, please feel free to go over Midway’s documentation or to try download it from here.
