Intuit Messaging Platform on Public Cloud — Scaling High and Beyond

Authors: Amit Jere, Madhavan Narayanan, Gokul Sarangapani and Jason Webb

Summary

Intuit Messaging Platform is a transactional, multi-tenant platform that powers many Intuit flagship products including Quickbooks Online, TurboTax Online, and Mint providing classic pub-sub and queuing use cases at an enterprise scale. Built on top of RedHat’s ActiveMQ, this platform operates in multiple public cloud regions. Starting in 2015, the platform began operating out of Intuit’s private data centers and successfully served customer needs. However, in 2018, significant degradation in performance and scalability started to occur, as a direct result of an increase in customer traffic and Intuit’s migration to the public cloud (AWS).

In this post, we discuss the problems and the solutions we arrived at to solve them to run a scalable, high-performance messaging system in the public cloud.

The Problem

We operated the platform in our data centers until 2018 and migrated to public cloud infrastructure (AWS) as part of our data center migration strategy. We observed significant degradation in performance and scalability as a result of a) migrating to the public cloud infrastructure and b) manifold increase in the load the platform supported.

The Solution

We focused on the following fundamental aspects:

1. Every broker in the Messaging system would need to be tuned for maximum read/write performance
2. The communication and efficiency across the brokers would need to be at its optimal level within and across different AWS regions
3. We needed to improve the scalability of message filtering and routing
4. Adding hardware to address scalability and performance concerns was not a viable option because of cost implications

Solving these issues required us to deep dive into the core of ActiveMQ, the underlying product for the Intuit Messaging Platform.

The Messaging team at Intuit came together to solve these performance and scalability issues to deliver multi-fold improvements compared to the hosted solution in Intuit private data center. The improvements in key performance indicators of the platform include a 4x increase in throughput, 3x improvement in latency, 3x better CPU usage, and a 30% reduction in hosting cost.

The Backstory

Intuit Messaging Platform started its journey as a simple messaging solution for a small set of customers in 2015. It evolved over the next few years into an enterprise messaging platform supporting all of Intuit, which included thousands of clients using the platform for asynchronous queuing use cases.

To support Intuit’s migration to AWS, we operated the Intuit Messaging Platform in a hybrid model, with brokers running in both Intuit data centers and in AWS. During this phase, we observed a significant difference in the performance of the ActiveMQ brokers running in our data centers vs. those in AWS. Specifically, we noticed a similar configuration in AWS lagged by a factor of up to 5x in throughput. When the majority of the traffic was generated by brokers running in our data centers, the performance of the public cloud infrastructure issue was not a major concern.

Intuit had an aggressive goal to vacate our data centers and move to the public cloud by the end of 2018. In order to meet this goal, we moved the Intuit Messaging Platform, including all brokers, to AWS and abandoned the hybrid model. This change in the operational model gave us an opportunity to critically evaluate the performance and scalability needs of the platform and find solutions to improve all bottlenecks.

We have rolled out these solutions to production in a phased manner over the last 12 months. All our learnings are captured in the white paper which covers various performance and scalability aspects, troubleshooting and experiments conducted, and the solutions designed to meet the business needs of high availability, performance, and scale.

At Intuit, we pride ourselves on creating a culture where our engineers are empowered to do the best work of their lives driven by Intuit’s operating values. We’re inspired to share our learnings with the community on how we achieved engineering and operational excellence at scale.