Three ways to aggregate microservices data for HK01 Newsfeed
Angus Tse, Director of Engineering at HK01
HK01 is a fast-growing Hong Kong digital media. It is the most popular Hong Kong digital media with five million monthly unique users, although the HK population is eight million. HK01 would like to extend the impacts in Hong Kong, transforming to Media+, providing lifestyles services to Hongkongers. We are introducing Lifestyles Services (Event Ticking, E-commerce .etc) to the AI-powered newsfeed, driving exposure to users. Since this newsfeed is the landing screen of the mobile App, a high scalability and reliability architecture are the primary concern.
Option 1: Pull data from lifestyles services via API call.
Using API calls to pull data from Lifestyles Services is the most straightforward method to achieve the integration. However, this solution may have scaling issues when spike traffic arrived. It is hard to guarantee downstream services scaling out/up on time. Lifestyles Services might be overloaded when requests spike came. Furthermore, a slow downstream lifestyles service may drag the throughput of the whole newsfeed. It is a so-called backpressure issue.
In this example, the Feed Service received 1000 requests per second (rps), but the downstream Service A response rate was just 800 rps, which means there is a 200 rps deficit. This deficit affects the user experience directly. End-users feel the mobile app loading speed is slow. It might drag the newsfeed throughput and cause the Feed Service (partial) outage in the worst case.
Control, Buffering, and Dropping are general solutions for backpressure issues [@jayphelps post]. However, these are not good options in the HK01 recommendation feed since this will significantly affect the user experience. Instead, a separation of concerns (SoC) architecture for isolating the peak traffic should be a more desirable direction.
Option 2: Push items to Feed service thru API
Lifestyles Services push new item/item update to feed inbound API. The inbound API performs data validation and then enqueues the item. The feed item processor handles items from the queue and stores the data eventually. Moreover, the outbound API is responsible for the 1,000 requests per second independently.
This approach isolated the traffic peak from other Lifestyles Services since there is no immediate dependency on them. Only the feed boundary needs to take care of the scaling and reliability issue for the peak traffic. Another upside is the standard inbound API facilitates new lifestyles integration, has a minimum dependency, and reduces development overheads. Alternatively, replacing the API call with a message queue also has the same advantages, but it isn’t easy to validate data with simple implementation.
Option 3: Push + Pull approach with Event Notification
The previous blog post mentioned we had applied the Event Notification pattern for decoupling our services. The Push + Pull approach, the Event Notification pattern, broadcasts something that happened in services. Similar to Option 2, This approach isolated incoming requests, but it required Feed Service to subscribe to Lifestyles Services event and call their API. Development and test effort are needed on both sides when a new service would like to appear in the Media+ feed.
Finally, we adopted Option 2, pushing items thru API for our Media+ newsfeed. This approach is separating the concerns and isolating the traffic. Simple integration thru standard restful API is a great advantage that reduces communication and development overhead.
Mostly, we don’t need rocket science or a tech stack to make stuff scalable. Rethinking the dependency, pull/push pattern, and eventual consistency are always your good friend.
Thanks for reading. If you enjoyed reading this article, feel free to support us by hitting the clap button 👏 and to help others finding this article. We are hiring, job descriptions can be found HERE. You are welcome to drop me a message if further questions.
