Enriching Streaming Game Events with Benthos and Apache Kafka: A Plarium Case Study

Introduction

In the fast-paced world of online gaming, capturing and processing game events efficiently and in real-time is crucial. At Plarium, a leading game development company, we faced the challenge of high throughput events enrichment while decreasing the game events footprint sent by the game app. We wanted to send very slim events, containing only relevant data to the event itself, and still get in real-time an enriched event with additional attributes like session and player-level information.

To address this, we implemented a robust and scalable solution leveraging Benthos and Apache Kafka, deployed on Google Kubernetes Engine (GKE). This article outlines the technical details and benefits of our solution.

The Challenge

Plarium’s gaming platforms generate a vast amount of events, including game logins, player actions, and in-game activities. The player login events contain player and session-related data, such as the game version and player registration date.

Other events such as battles should be kept as slim as possible. Having said that some of our streaming processes still require the extra player/session level data. So we had to enrich these events with the data from the login event. Sounds pretty simple right? But there are some additional considerations:

• Sequencing of events is not guaranteed — We might have a Late Login event, the rest of the session events can not be enriched at the time of their arrival and must be handled once we get the login event.
• Missing events — In case we did not get the login event after X time we still need to send the slim event as is.
• Configurable — The list of enrichment attributes should be configurable and easy to deploy (no new images)
• High throughput — The enrichment process must have very low latency
• Scalability — The solution must be scalable
• High availability — The service should have minimum downtime.
• Persistency — Zero data loss
• Memory performance — Keep Memory consumption efficient, We need to keep only the relevant player and session cached data.

Solution Overview

To meet the challenge, we devised a robust streaming solution that utilized Benthos as the data stream processor and Kafka as the message broker. Here’s a high-level overview of our solution:

1. Game Login Events Producer:
   The gaming platform generates game login events, which are sent to a Kafka topic as the entry point to the data streaming process.
2. Game Events Producer:
   The gaming platform generates player actions, and in-game activities events, which are sent to a Kafka topic per event type which is marked as a slim topic (prior to enrichment).
3. Benthos Deployment on GKE:
   We deployed Benthos on GKE, allowing for easy scalability and management of the stream processing pipeline. Benthos acted as the intermediary between the slim game topic and the enriched Kafka topic.
4. Enrichment Process:
   Benthos was configured to extract game login events from the Kafka topic and cache them in Redis (GCP Memorystore) with a defined TTL, they will be deleted once the session timeout is over. A second input was configured for all “slim” topics to be processed by the enrichment processor. For each “slim” event this processor fetched additional data from Redis cache, In case a relevant key was not found in the cache the event is sent to a retry topic with an additional retry counter attribute. The third input would be the retry topics which will be read from every defined interval. Once the events were enriched, Benthos forwarded the enriched events to another enriched events topic (per event type), ensuring the real-time flow of data to downstream systems.
5. Consumers:
   Downstream applications and systems consumed the enriched events from the Kafka topic. These systems utilized the enriched data for various purposes, such as personalized in-game content, analytics, and reporting.
6. Enrichment configuration:
   A deployment process using cloud build was created to read the configured enrichment attributes from a DB and dynamically create the Benthos configuration file.
7. Monitoring and Alerting:
   Using Prometheus® and Grafana®

Benothos Configuration sample:

Advantages of the Solution

Our chosen approach offered several significant advantages:

1. Real-Time Data Processing
   By leveraging Benthos and Kafka, we achieved real-time processing of game events.
2. Scalability
   The deployment of Benthos on GKE ensured horizontal scalability, meaning we could effortlessly handle fluctuations in event data volumes during peak gaming hours. This ensured a smooth and uninterrupted gaming experience for players without concerns about data bottlenecks.
3. Fault Tolerance
   Both Benthos and Kafka are designed to be fault-tolerant, ensuring that the game events processing pipeline remains highly available. In the event of component failures, the system would gracefully recover, preventing data loss and maintaining overall system stability.
4. Data Enrichment Flexibility
   Benthos provided flexibility in implementing the enrichment processor. We could easily modify or add new enrichment logic as per evolving business requirements without impacting the overall streaming process.
5. Simplified Management
   Using GKE to host Benthos reduced the operational overhead associated with managing infrastructure. This allowed our engineering teams to focus on developing game features and improving player experiences rather than managing infrastructure.

Conclusion

Plarium’s implementation of a game event enrichment solution using Benthos and Kafka on GKE proved to be a success. By augmenting game events with relevant information, we were able to deliver a more engaging and personalized gaming experience to our players and real-time information to our internal customers. The scalability, fault tolerance, and real-time capabilities of our solution ensured that our gaming platform could handle significant data volumes without compromising performance.

As Plarium continues to innovate and develop new games, this streaming solution lays a solid foundation for handling event data efficiently and opens doors to further enhancing our players’ gaming experiences.

References

1. Benthos documentations: https://www.benthos.dev/
2. Benthos enrichment example by Benthos: https://www.benthos.dev/cookbooks/enrichments/
3. More about our use of Benthos by our Benthos Master Almog Almado:
   https://medium.com/plarium-engineering/how-we-reduce-our-streaming-cost-by-90-using-benthos-fe04fcdffea7