Trell Cash: The new age social commerce gamification engine | Architecture Design
Welcome to the second post of Trell Cash, where will be finally talking about tech! How the product design transcended from Figma and Google Docs to Scalable pods on Kubernetes infrastructure.
Note, this post is one of the 2 blog posts regarding Trell cash, you can find them here :
- Product Requirements
- Architecture Design (You are here)
Without further ado, let’s jump right into the engineering design, (note, I will not be repeating what are the games and milestones here, for the basic product design recap, kindly go through the first blog in the links above)
The Best data structure? 😎
Trell cash is a gamification engine and being one, it will host all kinds of games and we will need to store various activities and checkpoints that yes, this user completed this checkpoint.
Every game is different right? Every different game can have different sorts of checkpoints. For example, your classic signup milestone can have simple true/false kinds of checkpoints…
Did the user enter their age group?
Did the user enter their gender?
Did the user follow some creators and so on…
These are all binary questions with simple True and False answers which can be easily answered.
Let’s take one more kind of game, you earn coins on how many followers you are getting / how many people are shopping via your affiliate link?
In this activity tracker, you are going to store who follows whom or who used this guy’s link in a separate table anyway, so why populate the common checkpoint activity table?
All these kind of data needs a lot of flexibility so that anything can be stored in the activity data.
The first thing which comes to mind?
JSON
Now, if JSON is the decided structure why not just use a No SQL database engine? Mongo is literally made for this job!
See, in this particular use case, we are looking at way more updates than reads and if there is one thing SQL can handle is quick inserts and data manipulation. As to address slow reads, a clever combination of userId + goalId index can be your savior.
Microservice Spider Web 🕸️
This little gamification system in itself is a small part of an architecture that supports a scale of 30M + DAUs and if you noticed the product requirements, this little gamification engine is buried deepest in the network requests (topologically speaking).
And of course, an engine of this scale does not solely rely on one microservice, the integrations with multiple user-facing services are an essential part of the stability of the entire net and sanctity of the reward credit system.
The user-facing systems interact with common load balancers to relay updates regarding the checkpoints completed by a user. The diagram below would give you proper insights into how it works.
What happens when you complete a milestone?? 💰
Enough talk on how everything is stored, let’s talk about what happens when you WIN!
User microservice notifying about the user update. 🏓
A simple hit on the relevant endpoint is all it needs to get the engine running on where the user stands and what all checkpoints need to be updated.
The verification and update strategies could be that either the user-facing services verify everything that yes, these entries are all correct and just give the green signal to the engine that yup, this milestone is completed.
OR
The gamification engine once again does a sanity check whether these values add up and whether this is an actual request by user microservice or an imposter? (Beware of master-slave replica lag here, they are a real bummer, kindly refer to the read-your-writes section here to understand what could go wrong)
Are all the checkpoints done? Time for KA-CHING! 💵
If all the checkpoints are cleared, we send the e-com microservice the go-ahead that yeah, this winner right here deserves a cookie, go prepare him some worth of 20/50/100/500/x depending on what he earned.
Handling high-velocity data for “Watch Videos” Milestone 🚀
We have a rather interesting engineering problem here when dealing with high-velocity data in the cases where we have to keep the count of videos watched for a particular user every day.
Your first guess would’ve been storing all this stuff in a database like SQL, right?
An average user will watch and scroll through some 5 videos in a minute, if we plan to store a data entry every time a user watches one video for an average of 1,000 users per minute scrolling through our content, It will average out to 5,000 inserts per minute and in peaks, it might hit 10k. So clearly database ain’t a good option if you like the sanity of your master database.
What’s the alternative? 🧑🏫
Redis! This high performant key-value store has a beautiful data structure called HyperLogLog which works just perfectly for our use case.
This will basically convert the operation of inserting a new row in a database to increment the counter for a key in a key-value store. Hence, a massive performance boost for your insertions.
basically, whenever a user consumes some content, the metadata is sent through to our content APIs and if sufficient conditions are matched, the video is logged into Redis HLL. Now whenever the client asks for the current video streak, all we gotta do is pull out some magical data from our goal activity entry and some data from an interesting Redis key 😉
This frees you from figuring out indexes in SQL and letting your cache layer do the computational hard work!
Scale? 🌙
I will not be going into the configurational details of this gamification engine pods, but to give an estimate about the scale this microservice can handle please refer to the following picture.
On average, there are 11 pods running for this service and the average traffic is about 36k requests every 5 minutes. Hence the current scale allows a throughput of 7000 2xxs per minute with a 5xx rate of ~0.02%.
Conclusion ✅
All in all, the gamification engine has been presented in its best state. Of course, there are multiple factors like pod scaling and CI/CD which have not been discussed here (configurations won’t be shared of course).
The entire microservice has been written in Go language and I would 100% recommend you to try writing your next backend service which requires multiple async processes to run simultaneously in go or rust!
The services look simple but when it comes to deadlines, the easiest of the integration nuances can get on your nerves so watch out for that!
Until next time 👋
Resources 🔗
- Redis Hyper Log Log
- Arpit Bhayani’s read-your-writes
- Horizontal Pod Scaling in K8s
