Building an Asynchronous Integration Layer to Salesforce with Heroku and Kafka

Introduction

At RadialSpark, we recently designed a solution for a customer of ours that conducts surveys and curates the results into a set of high-quality leads for their clients. Since avoiding cross-contamination of data is essential, this customer operates a unique Salesforce organization for each client.

While this separation of systems has allowed the firm to establish a highly customized lead-generation process and deep trust with each of its clients, it also presented challenges as the company tried to build a streamlined quality control (QC) process. Historically, the QC team switched between more than 30 unique Salesforce organizations to run QC processes on new leads. This created an administrative burden as the team lacked a centralized location for vetting leads.

Our team set out to build a system to standardize the QC process, one that could handle the data volumes of today and scale to meet a significant increase in data volumes that was expected in the near future.

For us, the key to developing a scalable solution was to create the QC application on Heroku. Building on Heroku decreased development time and allowed for streamlined integration with Salesforce through Heroku Connect and secure, simplified data storage in Heroku Postgres.

Heroku Connect transfers new lead data from each Salesforce organization into Heroku Postgres without hitting the transactional API limits set by Salesforce. However, Heroku Connect does not handle data transformation or normalization as the data model for each Salesforce organization has been uniquely customized to each client’s respective needs.

With the connection between the Salesforce organizations and Heroku handled by Heroku Connect, the remaining challenge was to transform the data so that the QC application could handle the superset of customizations from each organization. Apache Kafka provided an ideal solution for processing and normalizing data and for modifying schemas in the QC application in response to changes in Salesforce.

Transforming and normalizing data with Apache Kafka

Our team used Kafka to build an asynchronous integration layer that enables each Salesforce organization to publish data to a dedicated Kafka topic. Since each Salesforce organization is uniquely customized to meet the client’s needs, the lead data that flows from each Salesforce organization must be normalized to work within the QC app. Kafka supports these needed transformations.

The data flow is asynchronous to preserve the user experience. Each Salesforce organization pushes data as fast as needed, and the QC application processes it as fast as it can. If data flow must be slowed to preserve user experience (for example, the QC app might become sluggish or non-responsive because of background record processing), the process slows without breaking data flow.

Loose coupling enables each segment of the data flow to run at its own pace. Apache Kafka’s messaging capability enables the tracking of incoming messages and gives the receiving system time to read through the messages.

Tracking and modifying schema changes with Apache Kafka

To track new fields in each Salesforce organization and update how those fields are transformed and fed into the QC application, the team employed the Heroku Streaming Data Connector with Apache Kafka Connect.

Heroku Streaming Data Connectors are the perfect complement to Salesforce’s Heroku Connect for handling large volumes of transactional data. Heroku Connect provides a scalable solution for transporting large volumes of data to or from Salesforce organizations. Apache Kafka ensures scalability by buffering (or caching) the considerable volumes of data being transported between microservices developed on Heroku. Heroku Streaming Data Connectors provide a ready-made option for delivering data from Heroku Connect schemas on Heroku Postgres to Apache Kafka topics, bridging the two solutions. Streaming Data Connectors effectively provide change data capture (CDC) on Postgres tables and publish those change events to corresponding Kafka topics.

This ensures that the QC application effectively responds to each change and supports powerful management of unique metadata in each Salesforce instance.

Apache Kafka and Heroku work together to create a solution that not only handles the data loads of today but is built to scale to meet an expected increase of 25x in the near future.

Benefits of Apache Kafka

As the cornerstone of the solution, Apache Kafka enabled our RadialSpark developers to integrate and transform data in a scalable way. Without Apache Kafka, the team faced building a homegrown solution from scratch — a task that would have required a significant upfront investment of time and money, along with ongoing developer support and maintenance to onboard new Salesforce organizations for new clients.

With Heroku Connect, Heroku Streaming Data Connectors, and Apache Kafka underpinning the solution, the QC team now has a centralized system for distributed data points and a tool that has eliminated swivel chair integration from their daily work. Furthermore, the firm is able to onboard new customers without the support of a Salesforce SI partner or internal development team; all that is needed is an admin-level resource with some training and reference documentation. The entire team does their job in one spot, within the Heroku QC application, and that work affects each Salesforce organization.

Conclusion

In the not-too-distant future, most companies, not just those with an eight-figure annual revenue, will be grappling with the challenges of large data volumes. As Harvard Business Review notes, “companies in the top third of their industry in the use of data-driven decision making were, on average, 5% more productive and 6% more profitable than their competitors.”

Is your company managing your data in a scalable way that enables teams to make data-driven decisions and create efficient processes? If not, you may want to consider a solution based on Heroku and Apache Kafka on Heroku, which are the cornerstones of many big data management solutions and are especially well-suited to Salesforce-based organizations. Together, they provide the backbone for streamlined development that limits overall spend and results in a solution that is straightforward, scalable, and easy to maintain.

To learn more about the technologies covered in this post, check out:

• Heroku Connect
• Heroku Streaming Data Connectors
• Apache Kafka on Heroku
• Introducing Apache Kafka on Heroku: Event-Driven Architecture for the Cloud Era

About the authors

Justin Ludlow is a co-founder of RadialSpark, a Salesforce SI Partner specializing in cross-platform integrations and app development centered around Heroku. During his seven years working in the Salesforce ecosystem, he has been a developer, technical team lead, sales engineer, project manager, and technical architect, before stepping into the role of Chief Technology Officer at RadialSpark. As CTO, Justin designed the core technology stack used in most RadialSpark solutions. He is responsible for architecting custom technical and business process answers to his customers’, as well as RadialSpark’s internal, needs. His customers range from small businesses located across the street from his offices to household name-brands recognized globally. A self-proclaimed “puzzle solver”, Justin doesn’t consider himself a businessman, and his current “puzzle” is getting all of RadialSpark on Chromebooks to prove that a software development firm can be 100% cloud-based.
Email, LinkedIn

Ryan Cox is a Salesforce Distinguished Technical Architect supporting High Tech accounts in North America. He joined Salesforce three years ago after a long career at IBM primarily focused on implementing business process management and integration solutions.
Email, Twitter, LinkedIn