Building a Generic Enterprise Application Integration Pipeline

One pipeline to rule them all

At Whatfix, one of our solutions is to show context-aware and personalized help as an overlay on enterprise applications. Integrations with enterprise applications allow us to efficiently contextualize content for our users. Based on these integrations we are able to target information based on:

• User roles
• Survey responses
• User cohorts, etc.

In this post, we will discuss the high-level design and the mental-model of our generic integration pipeline to connect to enterprise applications to support a variety of use cases including paginated APIs and asynchronous APIs etc.

Integration Pipeline and Steps

Simple integration pipeline

Each integration pipeline can have multiple steps. Integration steps are of two types:

1. REST Connector: to call REST APIs of a system. Typical configurations for this step are URL to call, the HTTP request method, authentication details, request payload etc.
2. Transformation: to convert the response object of one generic REST connector into the request object of another generic REST connector using JavaScript.

Complexities of the Real World

There are many use cases which may not be as simple as the three-step pipeline and require more steps.

1. Pagination

Real-world APIs may be paginated. Pagination itself can be of many types, such as:

• Cursor based
• Offset based

For more details of the pagination, you can read the evolution of pagination at slack.

2. Asynchronous APIs

Systems with a large amount of data may optimize storage and compute by exposing APIs that may not return data instantaneously but the response contains a URL to poll for the status of the background job to fetch data. Once the job is complete, the job status call response contains the URL to be finally called to get data.

3. APIs returning links to be called further

Certain APIs return a list of URL links to be further called to get required data. In such cases, we need to iterate over the list of URLs and make calls to get the required data.

Supporting Pagination, Async APIs and URL links in the Integration Pipeline

To satisfy a wide variety of use cases with similar UX as of simple generic pipeline, we turbocharged our transformation step of the pipeline.

We defined two keywords nextLinks (a JavaScript array of objects) and data (a JavaScript object), which could be set in the transformation step.

We pick the nextLinks and run the previous REST Connector step again by taking the nextLinks as URL parameters. And data is always passed down further steps for processing.

Turbocharged transformation step

The condition to break looping from the transformation step would be when nextLinks is null.

For Async APIs, the transformation step could look as follows:

Next Steps

Now that we are done with building a generic integration pipeline we are working to build specific connectors for different enterprise applications.

This is the work of the Data Integration team at Whatfix. Thanks to the Data Integration team and Nibu Thomas for the review.