Creating a System for Tight Integrations via API & UI (aka Highways)
Ever waste time navigating back-and-forth between two different SaaS applications, even if they’re already integrated with each other? This is a problem we aimed to solve in Pipedrive. Let me give you a closer look into how we constructed a mechanism to bring our vendors (app partners) closer together through an easy integration process.
When I started working at Pipedrive 4 years ago, it was already praised for how well it integrates with other platforms. Within a couple of years of joining, our team built out a stable OAuth solution and a snazzy marketplace where we moved all of our pre-existing integrations to. This allows people to more easily find integrations/apps. Job’s done, time to celebrate 🍻(which we did).
By the end of the first iteration, we had created a pipeline for adding integrations in a faster and more refined way. Developers could gain access to our marketplace by request and we created examples as well as documentation about how to create an app. The introduction of our app catalog helped to make apps more visible to our end-users and thereby increase the number of app installations.
Although we gave the whole process of creating and displaying apps a revised look, it was still built upon just making requests to our public REST API. Despite there being nothing wrong with this, it can still create a mental border between integrations. We also received some feedback that hinted that having to switch context between platforms wasn't very user-friendly. By that time we started thinking of ways to bring integration features even closer to our users and established two main ways:
- App actions & panels
- App highways 🛣 (manifest)
App actions & panels — allows third-party vendors to show information regarding an object (such as deal or person) in a more visually pleasant way. App actions API offers the possibility to build forms inside the Pipedrive UI.
App highways — An app can register itself as a specific provider, in this case, it gains access to hidden APIs and gets called when events happen inside Pipedrive.
Although both of these ideas have a similar underlying workflow, in this article we will focus on the manifest side (panels and actions really deserve their own article📝). With the most common integrations, the data between two apps flows only one way (usually). Users either export data from our side to be processed on a third-party application, or they import some additional data that is processed from outside Pipedrive. So where does the manifest come in?
Highway for invoices
As an example, let’s use invoices. After an app developer has created their app, they can upload a JSON-based manifest file to be included. This file will act as an interface between the Pipedrive platform and another SaaS solution.
Afterward, users of Pipedrive will see different options for who they want their invoice provider to be:
Once a user chooses an app, we can then grant the end-user extra ways to interact with the integration. In our invoice example, we allow the user to create invoices and view the status of invoices that are attached to a given deal.
In reality, it doesn’t matter who the provider is, we just send out user events and make requests based on the integration that was installed.
How does it work under the hood?
When vendors are creating an app, we give them an option to upload a JSON file which acts as the glue between our API and their integration. You can find an example of how one looks like on our Developer site. The parsing, validation, and usage of that uploaded JSON are up to our marketplace.
Once a valid manifest is present for an app, other hidden APIs and services inside Pipedrive will be available to access it. Based on what the vendor adds to the manifest, we know what endpoints and features to call.
In essence, manifest is just a set of rules and endpoints that we require from applications that want to integrate with our platform.
- The vendor sets up their app as a service provider and includes a valid manifest file
- Pipedrive's internal APIs and event flow grab the user-selected external platform
- Based on the events or actions, we make requests to URL’s provided in the vendor manifest, with the predefined structure
Technical hurdles
As each app is different, one of the issues we encountered was that we had to keep a balance between generic API consumption and simplicity. For example, with invoices, it should be mandatory to be able to create invoices (duh), but one of the platforms had an option to export invoices using a template.
We solved this issue by introducing a “features” property that marks if the invoice app was capable of handling exports using different kinds of templates ✅.
What about webhooks 🤔
You might be thinking that this all sounds very similar to the concept of webhooks, as they also send out events based on user activity.
The core difference here is that we allow data to flow in both directions, a user can request data from other platforms to be shown, but they can also make secure requests to other platforms from the same Pipedrive UI.
In addition, webhooks have more general events (e.g. Deal added) while a manifest grants access to much more granular user events (e.g. deal opened).
Conclusion
I personally like the new way we are handling a tighter third-party integrations system and I hope we can build upon this more. I have heard that some platforms allow injecting iframes, but at the moment this sounds too wild for us.
If you want to learn more about the highways, we have two public ones you can read about and try out for yourself — 1 for invoices and 1 for video-calls. If you have any cool ideas regarding this, then I would be happy to hear your approach on the matter.
