How we leverage Elastic Reverse Search (Percolate) to power our coupons system
Every C2C Company (Company dealing with customers directly), at some point, has to introduce coupons/discounts, to enable companies to penetrate their product to a larger group of audience restricted by budgets, and Doubtnut being completely user-centric had to implement the discounting service to be able to help a lot bigger segment of society. Day to Day best example for coupons would be Amazon, Amazon offers various kinds of coupons: discount based, cashback-based, offers on purchasing other services.
In growing startups such as Doubtnut, delivering the task in a short period is the priority, as also said by Mark Zuckerberg in the initial stages of Facebook his ideology was to move fast and break things i.e. providing the user the new functionality ASAP and not worrying about breaking the flow. Similarly in Doubtnut, there are a lot of inventions both product and engineering going on and for this we follow an MVP approach, where we start with something small, see how it’s impacting the users or the business and if there are positive data around it, we ramp up on the same. On similar lines, the first Coupon Architecture was built to meet the needs of a small user base. The structure was roughly as below:
Target Groups: Target Group is used for bucketing a user into a different set. This is a subset of behavioural segmentation which is done by many brands to customise the experience according to each segment.
Reason to revisit:
- High Latency: The first version of the API had a response time of 400ms — barely qualifying the standards (100ms), but as the service turned out to be a success and enabled an easy way to categorise users and provide them additional benefits, a lot of new campaigns were launched to attract users and our data grew exponentially resulting in API response time around 3.5sec — 4sec. This was a criminal offence! The service which was designed to help users purchase the product while not overstepping their budget limits was now the reason for the bad user experience during buying.
- Need for higher throughput: In the first version of the service the coupons were visible quite late in the process, hence the API throughput was low. However, given that it had a lot of use cases that could be used in high throughput assets (like the homepage), we had to reduce the response time and resource consumption. It wasn’t just sanity, it was necessary to enable use cases. So to fix this service we started looking out for ways to structure the above data. What we wanted was — a way to reverse search coupons with the users. As in — if there is a way to define the properties of a coupon, and whenever we need to find the applicable coupons for a user, we could do that.
ElasticSearch Percolate to the rescue: ElasticSearch was already in use in our company on a large scale for our core product — i.e. questions and answers and calculating other statistics on the fly. ElasticSearch has a percolate API, which does exactly the above. It’s the reverse of the classic ElasticSearch use case.
Detailed doc on how ElasticSearch Percolate works, through examples: https://spinscale.de/posts/2021-09-15-understanding-elasticsearch-percolate-query.html
In our case, the queries represent the coupons, which represent the filters based on user properties. A user’s doc is then created at runtime, fetches all the coupons (filters) applicable to the user from ES and after that, the discount computations are done based on various factors.
Coupon Service: To further speed up the process and as a part of keep learning and trying out new techs, we built the coupon service in GOlang (common tech stacks used at Doubtnut for backend: Node.js, typescript). Similar to Node.js promises, GOlang provides goroutine which is an execution thread running concurrently with the rest of the program, and since there were a lot of async tasks in computing the applicable coupons, we used GOlang’s goroutine extensively to keep the API response time to a minimum.
Performance Testing:
p50 and p90 latency comparison for both implementations
Future Plans:
We knew we built a service that could filter out millions of students at runtime and with very low latency and very high throughput, we call it a Target Group Service. A very powerful service that could solve multiple use cases across Doubtnut. The biggest of them was Notification Service. Notification service also had the same use case wherein we wanted to target a specific set of students for every notification. We still see such use cases in a lot of modules in Doubtnut and we are trying to extend this architecture in all of them.
