Broadcast Server(gen-server) using Erlang

Why the Broadcast server?

“Everybody wants a cake but nobody wants to bake.” This is one of the perks of being an inventory platform, every sub-system requires a subset of particular data, be it OLAP, SEO or any other team empowering buyers/sellers through the different Business Intelligence(BI) tools.

As data is continuously growing rapidly, ETL operation is not feasible frequently and also not advisable on the production database. ETL operation will eventually lead to inconsistency(stale) of data available for all the other important sub-systems. Also, ETL is not a scalable solution.

This system helps to provide efficient and consistent real-time data to all the sub-systems.

How does the Broadcast server work?

Following is the diagram of the underlying system architecture, System architecture

System Architecture

• All the applications (Inventory platform, broadcast server, OLAP, SEO, etc) are in Erlang and DB are Mysql.
• Whenever an update happens on the inventory platform, the same inventory details will be passed to a broadcasting server using an asynchronous call.
• The broadcasting server will pass the data to all the sub-systems registered/listening during that time.
• On receiving data each sub-system will process the data as per their use cases and store data in the database(Mysql) so that they can serve later on.
• In Erlang, we keep each sub-system as a different gen-server on the same server/system.
• Each sub-system can register/de-register to get/stop the real-time data feed on run-time using POST API. Post body for register/de-register,

{
 "poolname": "process_pool_name/PID"
}

• Processing for each sub-system happens simultaneously using the efficient concurrency of Erlang.
• It is not duplicate data in each sub-system, each sub-system keeps the abstract data as per their use-case.
• The number of updates happens on normal off-season day, it can be 2.5 to 3x on a peak day. See Number of Updates per the Minute chart below,

Number of Updates per Minute

What does the Broadcast server achieve?

• It provides a scalable solution to sub-systems for realtime data and avoids doing costly ETL operations on the production database, as we at redBus_Blog use the direct database for search, not cache.
• Provide real-time consistent and reliable data to OLAP, SEO, and BI tools based on inventory data which is not possible using ETL.
• One of the use-cases is to provide real-time inventory data reports through a dashboard to the supply team.

References

• [Online]. Available: http://www.erlang.org.
• [Online]. Available: https://learnyousomeerlang.com/content