Automated Testing of Real Time Bidding Engine
RevX is a mobile performance technology platform that participates in close to 20B+ programmatic ad auctions on a daily basis. For every auction, the platform evaluates complex targeting rules and budget constraints across all active campaigns and finally choses a single ad for which a bid is placed.
Engineering and testing such a complex system presents many challenges. RevX engineering leverages automated testing to continuously verify the functionality of mission critical components like bidding engine, enabling us to build and release new features quickly with high quality. This blog gives an overview of how we automate testing of real-time bidding engine.
Let’s take a simple use case of testing the functionality of limiting daily ad exposure to a user.
Setup
A campaign is set up to limit total ads shown to a user to a maximum of 10 impressions/day.
Test Assertion
RevX should stop bidding for a user if it has already served 10 ads in the last 24 hours. In order to test this scenario, we need to follow these steps:
1. Data setup
a. Setup campaign with a cap of 10 ad impressions/user/day
b. Add creatives to the campaign
c. Target the campaign to a particular user id
2. Test Execution
a. Send a bid request for the user to the auction engine
b. Simulate auction win and generate an ad impression hit
c. Repeat step a & b 10 times
d. Send 11th bid request for user and verify the bid response
3. Verification
a. Check that we bid for the user for the first 10 times
b. Validate that the auction engine does not bid on the 11th request, and log the reason for not (e.g. F-Cap Reached) bidding for reporting & analytics.
As can be seen, testing a simple scenario like the one described above requires multiple steps starting from data generation, test execution to final validation. A test framework is expected to simulate user behavior (e.g. see X number of ads) and RTB request/response protocols. Furthermore, the real-time nature of bidding engine adds more complexity as the automation needs to verify the strict performance metrics around latency and QPS
To save time and make the process more reliable, we automate all the business logic using a test framework.
Automation
We use TestNG for automating our test cases and group them into various test suites. TestNG is the preferred automation tool for a lot of QA teams as it is designed to cover all types of testing use cases such as unit, functional, integration testing and more. It also generates reports in various formats and can be easily integrated with continuous integration tool like Jenkins.
We divide each step in the test process into separate actions. For example:
Data Preparation: Generating necessary data for executing the test case. This involves syncing with multiple databases (MySQL, Aerospike) that store campaign and user information.
Test Execution: Simulates auction events like bid request, bid response, ad impression tracking etc. This involves writing test code to interact with various components of the auction engine and ad serving system.
Verification: Validates assertions after test execution. This step involves analyzing test responses, generating logs and verifying final database state against expected values. For example, in the scenario described above, automation will verify that the user was served 10 impressions if there were 10 or more opportunities, and less than 10 impressions if opportunities were less. If the automation identifies that a user was served less than 10 ads, is that due to code bug or lack of opportunities.
The diagram below illustrates data flow and interactions between test framework and test environment.
RevX engineering team follows agile processes to be able to deliver code deployments on a weekly basis. Jenkins downloads the entire automation project from Github and triggers the TestNG main class to execute all test cases. It finally displays the failure/success report on UI and sends failure alerts to configured email addresses.
In a complex platform like RevX that power billions of auctions and millions of personalized ads daily, even a minor bug/downtime can result in huge performance and revenue impact. By leveraging state of the art automation technology, we can deliver new features in an agile way with extremely high quality.
Written by: Manhar Sapariya
About RevX
We are an app retargeting platform that powers growth for mobile businesses through dynamic retargeting. The platform is built on integrated and transparent technology combining four key pillars — audience intelligence, programmatic media, personalized ads, and ROI optimization. Mobile marketers across verticals like e-Commerce, travel, lifestyle, hyperlocal and gaming use RevX to enhance user engagement by activating new users, retaining existing users and converting existing users.
If you have any doubts, queries or ideas, do reach out to us at marketing@revx.io
