The New Salesforce Event Bus
Publish-subscribe (pub/sub) patterns and event-driven architectures are a popular choice for Salesforce and our customers to create dynamic, flexible solutions that perform at scale. Over the last six years, we’ve seen tremendous growth in these patterns. In 2020 alone, the Salesforce Event Bus saw monthly publishes increase by 237% and subscriptions increase by 1,842%. The Salesforce Event Bus is a multi-tenant eventing service built on Apache Kafka. Customers are using platform events and Change Data Capture for everything from offline sync for mobile apps to integrating external chat bots with Service Cloud.
Within Salesforce, we use platform events and Change Data Capture to build new products and integrate our tech stacks. The real-time integrations that events offer have been essential to building products like Quip, Einstein Prediction Builder, and Real-Time Event Monitoring.
Time for a new bus
All this growth has has led Salesforce to rethink the runtime for platform events and Change Data Capture. These products have been operating at scale for the last six years as a part of the core Salesforce platform. That means the Salesforce Event Bus has shared the same infrastructure as products like Sales Cloud and Service Cloud while powering pub/sub for all Salesforce clouds. This architecture has been extremely successful, but as Salesforce has grown and acquired new companies like MuleSoft and Tableau, the time has come for a change.
This is part of why we’re introducing a new, independent runtime for Salesforce Event Bus. This new Event Bus Service is internal to Salesforce and sits outside of the servers powering Sales Cloud and Service Cloud. By having an independent runtime, we can easily scale the service to meet the needs of Salesforce customers and entire new clouds. We can also adopt a microservice architecture for eventing that enables a stronger focus on the needs of Salesforce clouds and customers. Building a new service is also a chance for the eventing teams at Salesforce to redesign the runtime with the hindsight that comes from years of running a production service.
Customer experience
So, the most important question is: What does this mean for customers? Since the new event Bus is an internal service, much of this work won’t be immediately visible to customers using platform events or Change Data Capture. But, these changes under the hood pave the way for greater scale, performance improvements, and new ways to do pub/sub. One of the most exciting changes in this space is the forthcoming Pub/Sub API.
For Salesforce, the new Event Bus is a key part of how we’ll deliver new cross-cloud customer experiences. Instead of worrying about integrating Salesforce clouds and stacks, our developers can think about building products that help customers meet challenges at any scale.
Reliable rollout
With the new runtime set to come on line in the summer of 2021, the question is: How do we move billions of events to a new runtime without a failure? Thousands of customers integrate their systems with events or use products built with events. Salesforce has had to design a rollout that works at scale and keeps reliability and customer trust at the forefront.
A three-part strategy based on visibility, testing, and redundancy will guide the transition to the new runtime.
The first part is all about visibility. We expanded the internal metrics, dashboards, and alerting our team uses to provide a fuller picture of how the service works and give our engineers real-time notifications they can act on. We’ve also implemented synthetic probing, which tests the system in production by periodically generating artificial events to make sure everything is working. In the future, we’ll be exploring how to extend more of these monitoring tools to Salesforce customers.
The second part is all about testing. Salesforce engineers have performed extensive performance tests and a series of resiliency evaluations to verify how the Event Bus runtime behaves in production.
Finally, the third part is all about redundancy. Instead of making a hard cut-over to the new runtime, we’re taking a gradual approach. As we start shifting traffic onto the new runtime, we can seamlessly revert to our tried-and-true runtime if there’s ever an issue.
Introducing the Pub/Sub API
In addition to rolling out a new runtime for the Event Bus, Salesforce will also be introducing a new Pub/Sub API. This new API combines the power of the new runtime with publish and subscribe in one interface. It also provides client-driven flow control for event delivery. As a gRPC (Google Remote Procedure Call) API, the Pub/Sub API can be used with 11 different coding languages. Beyond making it easier to integrate Salesforce with external systems, we want to offer an API that’s tailored to pub/sub and builds on our experience with the Streaming API. The Pub/Sub API is slated for customer pilot in late summer 2021. For details on participating in the pilot, reach out to your Salesforce account team.
