Large Data Volumes in Salesforce — a new approach for 2021
Originally published at omnata.com/blog
Large data volumes in Salesforce can be challenging and there is a bunch of advice circulating on how to handle it best.
According to Salesforce, ‘large data volumes’ is defined as:
“an imprecise, elastic term. If your deployment has tens of thousands of users, tens of millions of records, or hundreds of gigabytes of total record storage, you have a large data volume.”
By the standards of modern data architectures, ‘tens of millions of records’ would hardly classify as large, but for Salesforce and many cloud applications, storing and processing big data was simply never intended to be a core capability.
This article will talk about large data volumes from across an enterprise, not data that has grown organically in Salesforce like having lots of customer accounts.
Two key architectural trends are happening:
- Companies are centralising data in cloud data warehouses like Snowflake and BigQuery.
- Platforms like Salesforce are systems of action, but they are no longer the system of record for all customer data.
In practical terms, if I use Service Cloud for customer support and run a high volume business with a hundred thousand customers, I’m not loading every single customer I’ve ever interacted with into Salesforce. They only become a record once they need support. It’s the system of action for my service agents.
The system of record has become the data warehouse. Containing a modelled view of each customer that combines all of their touchpoints across systems. This single view is used all across the business by people and applications. My service agent needs access to this to understand things like product licenses, usage, transactions, and contact history with other teams.
What does that change?
Considering the trends above and the following flow diagram for handling large data volumes in Salesforce, companies with modern data platforms increasingly fall into the ‘NO + NO’ path, which leads us to Salesforce Connect.
Salesforce Connect is often recommended for org to org connections or creating single-purpose integrations to other systems, usually via Odata or middleware like Mulesoft or Jitterbit. It isn’t a new feature, but it hasn’t been leveraged as an agile data connection before.
A new way to use Salesforce Connect
Omnata allows you to directly connect Salesforce to cloud data warehouses; Snowflake and BigQuery. These platforms are designed to handle operational scale workloads and using live-queries solves a bunch of problems faced by traditional extract and load integrations. Using Omnata unlocks integration use cases like product data, revenue and transactional data, machine learning outputs, and patient records.
No limits on dataset size
Omnata allows you to query from production tables in Snowflake or BigQuery that could contain millions or billions of records. Live-query removes the limits of storage and API quota and doesn’t bog down performance.
No longer maintain copies in Salesforce
The integrity of the copy or handling load and update failures cease to be problems to deal with.
Real-time data freshness
Live-queries mean data is always up to date.
Leverage the source of truth
Your data team are already centralizing and modelling all enterprise data so there’s no need to redo this work by integrating point-to-point.
Secure connection directly to your warehouse
Omnata runs in your Salesforce environment and there is no separately hosted service handling your data or credentials. From a security point of view, this significantly reduces your attack surface and makes it straightforward for initial security reviews and ongoing vendor assurance activities. From a privacy point of view, it removes the need for yet another third party data processor you need to trust.
Enabling Salesforce teams to do more
Salesforce admins and architects now have the ability to leverage all enterprise data in external objects. The best part of external objects is that data appears natively and can be used in platform features like reporting and automation. On top of this, Omnata makes big improvements to the out-of-the-box functionality of Salesforce Connect so that it’s far more user friendly and flexible. All of this helps the CRM team move faster and create a more valuable and enriched workspace.
Automates external object creation
Save hours of setup time defining data types and swapping between systems with a user-friendly setup experience.
Simplifies relationship management
Easily create parent-child relationships, rename fields for usability and create drillable object relationships.
Handles underlying schema changes
A single purpose integration breaks whenever the underlying schema changes, and this can happen a lot with agile deployments. Omnata gracefully handles changes instead of starting from scratch.
Transform Salesforce into a data driven system of action
The combination of Omnata and Salesforce Connect can help you rethink your large data volume architecture. It is simple, agile and suits the shift in enterprise data architectures. New data warehouses are fast, scalable, and reliable to support operational workloads. Omnata builds the critical link between your new system of record and primary system of action.
