The #1 Problem with Webhooks in Data Integration, and How We’ve Fixed It
Over the past year, we’ve talked to hundreds of customers and prospects and they were keen to share their many hurdles integrating their different services together. They’ve shared a very common problem:
The data their teams need to work in one tool, almost always exists in another tool.
This is a huge problem that companies try to solve with many different solutions — ranging from Excel and CSV exports, to Workflow tools, all the way to full-blown enterprise ETL solutions, or worse, homegrown solutions that set in stone their stacks and require constant engineering time.
We all know how large the Martech space is — thousands of apps, and all different.
The more we listened to you, the more we realized there was no way any company could ever keep up with the thousands of scenarios that were coming to us.
So, we sat down and thought about it really hard. We came up with a bunch of solutions that made it faster to build our connectors so we could therefore, deliver faster.
At first, we thought of making all decisions for the customer to standardize and simplify code and streamline it. We quickly realized Marketing and Sales stacks are like snowflakes. No two are alike, and each needs a very specific way of handling data.
Even these productive, streamlined approaches were not enough. We needed a different approach.
Building the Case for Webhooks
We realized we couldn’t really brute-force our way through the problem. Webhooks quickly came to mind since they are the lingua franca to send updates from one system to another.
There are tons of different ways services are able to capture webhooks, but we noted two things:
- Too limited. The methods exposed by existing tools to process webhooks data are always constraining in how they can capture the data and what they can do with it, or
- Too complex, offering huge screens of configuration to attempt to make it simpler, only making it more tedious.
We needed to think out of the box. Solve the problem in a different way.
Logic is Power
Then one day, we realized we already had a better solution all along. We already faced this problem when trying to build logic into Hull to compute attributes, transform customer data in realtime for our customers.
At the time, we were faced with building smart fallbacks for the company name that is attached as a user attribute.
Prototyping the UI to handle this gracefully was the equivalent of a living nightmare. Imagine building screens full of boxes, drop downs, and tiny controls to enable the user to add logic based on the attributes.
Recognizing the challenge, thinking about it really hard, we had a moment.
“This would be so easy if we could do it in Javascript….”
The Lightbulb Moment
Why couldn’t we let our customers write their own logic in javascript instead of constraining them to dumbed-down logic ? or forcing them through an arcane, proprietary ETL scripting language ?
See — our customers are pretty technical, and they more often than not understand and are able to write some Javascript, even if they’re not full-time engineers.
When you know how to write even simple code, the most complex interfaces can be summarized in a couple of lines of simple, vanilla code.
So we did that for our Processor connector.
And now, we just realized we could do the same to capture incoming data from webhooks and process it on the fly. This changed everything.
Why force data to go through point-n-click UIs when you can leverage the power of an amazingly well known language such as Javascript ?
What if we could add logic to all of this power?
You see, one of our most used and loved connectors is the Hull Processor.
It allows you to write very simple Javascript code, requires no complex framework, and lets you apply your exact logic to capture and map the data at web scale.
One of the biggest drawbacks to applications with webhooks in general is the inability to apply logic to data upon receiving it and sending it.
We applied this concept to webhooks, and the result was the Incoming Webhooks connector.
- Send us any data using webhooks in any shape, any form.
- Write a few lines of really simple Javascript to transform and shape the data exactly how you want, and associate it to the users you want
- Done, that’s it! We handle scaling, security, logging and everything else.
Suddenly, Hull is open to thousands of tools, and offers a very simple, innovative and more importantly infinitely flexible solution to integrating any source of data in your customer data profiles.
A Real-World Example
But instead of just telling you about it, I want to actually show you.
One of our clients wanted to record submissions from their Unbounce landing pages.
They’re using Hull to combine all of their customer data into one single source of truth and then act on their truth, but a missing link was always external landing pages.
Most basic marketing automation platforms can track the visit to the landing page, but it can’t track user events on the page. It also has a difficult time consolidating any new information obtained from the form.
But here’s the catch — the landing page service itself might not have the best integration opportunities either.
This is where the Incoming Webhooks Connector comes in.
In this case, we’re using Unbounce.
When a user converts on a form from Unbounce, it sends out this payload of JSON data to a URL of choice:
Now, we didn’t have a way to track form submissions on the client side.
To make sure we didn’t have any blockers, we registered a webhook that would ping the Incoming Webhook Connector each time a form was submitted.
It would inject the cookie-tracking identifier as a form field in the form — and then the webhook would ping us with the content of the form and the value we injected client-side.
Remember when I said you can apply logic to the data? Through the Hull Incoming Webhooks Connector, you can add in custom logic to take that entire payload and do whatever you want with it.
Here’s an example of how you can map that data to users:
The output is either logs for the debugger, events, or attributes you want to see, but the result is your data exactly where you want it, and how you want it.
Control + Flexibility
The goal of the Incoming Webhooks connector is to provide users with complete control and flexibility with their data, and support thousands of apps out of the box, with a simple and elegant solution that doesn’t constrain users in any way.
The result?
A way to do exactly that — apply custom logic to data flows in realtime. The possibilities are endless. You can see it for yourself, and we’re happy to help you build your dream integration scenarios.
I like crazy stuff that just works, Incoming webhooks are one of the coolest ideas we’ve had and we’re psyched to see the crazy scenarios our customers have started building with it, and we’d love to discuss yours!
