Reverse ETL : Opening the Floodgates for Operational Analytics

Competition in today’s world is increasing — Rapidly and Dramatically. This has meant that the Sales, Marketing and Support teams in every organization, irrespective of the verticals they are in, are having their work cut out to convert their leads faster and retain the right set of customers to meet the more often than not ambitious targets assigned to them.

This is where the emergence of data can open up numerous opportunities for them. Imagine the Sales team of a B2B technology firm having access to the Lead Quality Score or Conversion Propensity (based on certain attributes like company size, previous history, location etc.). It will make their lives so much easier to strategically target the top ones rather than going after all the prospects in their list. Or think of the Marketing team of an ecommerce retailer having near real time data of the Cart Abandoners and the product they were browsing on their CRM tools or Ads platforms. They can run remarketing campaigns to convert this set of high intent customers with an additional coupon offer within the next few hours. Or picture the Support associates being able to prioritize the tickets of their most loyal customers first & drive their retention better.

As data becomes the new oil, the organization which cracks this operational analytics part and empowers their on-the-ground teams with the insights and recommendations from the data (where it is needed the most), will win this competitive war hands down!

Emergence of Cloud Data Warehouses

Emergence of Cloud Data warehouses and DWaaS (Data Warehouse as a Service) paradigm has provided businesses with necessary horsepower to make the best use of their data. Unlike old times, this has meant that the data warehouses no longer require heavy upfront and maintenance costs, can be scaled up or down based on business requirements and the parallel processing ability opens up a plethora of opportunities in the field of real time analytics and artificial intelligence.

And understandably, this has also resulted in the data leaders of today rushing to jump on the cloud data warehouse bandwagon. Snowflake Inc’s $3 billion IPO in September 2020 is a strong testament to this excitement in the industry. In fact, the DWaaS market alone is predicted to reach around $25 billion before the end of this decade growing at a CAGR of 30% during the forecast period.

Data Mesh of Today’s organizations

So far so good. But given the data warehouses of today are so powerful, are the organizations prepared enough to have them as the “single source of truth” for all the data-based decision making? Do the Sales and Marketing teams have on-demand and near real time access to customer or prospect data to target and support their customers better?

The answer, unfortunately, is a NO.

While the Cloud warehouses have replaced the traditional warehouses in the new data architecture, we see many businesses are still heavily reliant on legacy connections they have built between their apps, often bypassing their warehouses, to drive these operational analytics use cases (see Figure 1). These connections were built in an ad-hoc manner by the engineering teams as and when requested by the business teams. As a result, they are often leaky, unreliable and take hours to transfer even a few thousand records. Even worse, some organizations depend on manual downloads and uploads of CSVs or text files to move data across apps, engaging multiple analysts in the process. Subsequently, this implies that most of the campaigns executed by the teams are run on stale data, need the same manual effort every time and give lower than expected returns.

While these ad-hoc or manual connections made sense earlier as querying the traditional warehouse was expensive, the power of today’s cloud warehouse makes this architecture completely ineffective and archaic. If the organizations don’t embrace the power of modern warehouses and continue to rely on the Data Mesh they have created over years, they will continue to be plagued by issues of data inconsistencies and lack of data democratization. This, as said enough, is something they can ill afford in today’s age of a highly competitive market.

Figure 1 : Data Mesh created by organizations over years

DWs as “Single Source of Truth” — Reverse ETL as the missing piece in Data Puzzle

Figure 2 : Data Warehouse as Single Source of Truth across the organization

So, how do the data leaders get it right? Understandably, Cloud Data Warehouses must take the center stage to enable the organizations to get their operational analytics right.

The ETL/ELT platforms like Fivetran, Airflow, dbT have established themselves as standard solutions while moving data from source systems to data warehouses. To complete this cycle, the data leaders have to start thinking of Reverse ETL solutions in conjunction to unlock the right potential of their data warehouse implementations. While app to app integrations can help the businesses move raw data, warehouse to app pipelines can help them sync insights, model recommendations and much more to really open a tremendous number of use cases for their operational teams (more about the use cases in the next blog). Organizations that can react faster to the customer demands will have the edge in this age and a Reverse ETL solution can definitely accentuate their ability to experiment, learn and set them up for success.

While many will argue for in-house engineering teams to pick this up as part of their roadmap, an out of the box solution (just like the ELT/ETL ones) is critical given the organizations have to act fast on establishing their lead via operational analytics and building production-ready connections between apps can easily take them many a quarters. In addition, since companies rely on multiple apps across teams these days, it only translates into significantly high engineering efforts to build and maintain pipelines specific to each one. Further, the same engineering bandwidth can be repurposed to improve the product features, which can definitely be a game changer for the company in the long run.

Castled and Reverse ETL

Castled is a cloud-native platform that enables Reverse ETL by helping businesses periodically sync their data & insights to their operational tools reducing the dependency on the Data & Engineering teams. Castled supports the major cloud data warehouses including Snowflake, Google BigQuery, Amazon Redshift and provides pre-built connectors to major apps like Salesforce, HubSpot and Intercom. The list of apps supported is only going to grow in the next few months.

Castled has an intuitive GUI which makes creating and managing pipelines extremely simple for anyone with knowledge of SQL scripting and basic understanding of the data schemas. It is highly scalable and optimized for transferring a large volume of records within a few minutes. And if you are an organization worried about the security of your data while using third party solutions, Castled follows all the standard security protocols to ensure that the data is encrypted the moment it leaves the warehouse until it reaches the destination tools.

You can visit Castled at https://castled.io/ and sign up to try the platform for free. And, to know more about the platform, book a free demo here.

Update: Castled has pivoted out of a Reverse ETL solution to provide a better way to solve the same problem. Castled is a Warehouse-Native Customer Engagement Platform built natively on top of the cloud data warehouses like Snowflake, BigQuery, Redshift, etc. Please read our blog to understand why we made this pivot.