Uplift | Failsafe HTTP Provider & Rule Based API Engine

Overview : In a microservice architecture, there are huge number of services communicating with each other via rest apis over network. This creates a higher chance for network or application level issues such that the service itself may also be unavailable temporarily. This is where NoBroker Uplift comes into play. This system is aimed towards providing graceful and fault tolerant retry capability alongside rule based conditional API support.

Goal : The aim of this system is to provide a fail-safe guarantee for any rest-api call or interdependent set of apis (with JSON path rules) via a retry mechanism upto a certain threshold limit.

Architecture : In order to ensure that this system is highly available, we decided to utilize google cloud functions for their scaling and auto-recovery capabilities. For triggering action for an accepted request, firestore on-create-http-listener is utilized by storing incoming http requests. Let’s understand with a step-by-step workflow.

Workflow :

a) Client makes an http request to uplift cloud function component [1].
b) On receiving a request for processing [2]; it checks if that is rule based call, it matches with conditions [2.11] and stores in google firestore [2.12] & returns a successful acceptance [3].
c) In case of direct api calls, it directly stores in firestore [2.21] & returns successful acceptance.
d) Now the request present in firestore is subscribed via on-create listener for processing and api call is attempted [4]. If that call is successful, the flow is terminated & request is deleted [5] or else the same is stored in firestore [5.1].
e) For reprocessing requests in firestore, an api is exposed for cron processing. This can be managed via cloud scheduler where you can configure it to run very 10 minutes or according to your requirements. That provides a list of n pending requests which are to be retried [6].
f) Each request for retry checks for a current retry count [7], if that threshold is met; this data is permanently pushed into failure store [8]. Or else, the same request is repushed into firestore for processing & entry is deleted from pending requests from the firestore table [9] .
g) This process is repeated for all requests to be processed by uplift.

Using uplift : Let’s finally discuss how the uplift requests would look like.

a) Rule Based Processing : Below is a sample of JSON path rule configuration. We are utilizing JSON path plus NPM library which supports regex matching as well.

[
  {
    "condition": "$[?(@.key && @.key.match(/TION$/i))]",
    "url": "${url}",
    "maxRetry": 2,
    "timeoutInSec": 60,
    "retrigger": false
  },
  {
    "condition": "$[?(@.key && @.key.match(/Test/g))]",
    "url": "${url}",
    "maxRetry": 2,
    "timeoutInSec": 60,
    "retrigger": false
  },
  {
    "condition": "$[?(@.key && @.key === 'TEST')]",
    "url": "${url}",
    "maxRetry": 4,
    "timeoutInSec": 60,
    "retrigger": false
  }
]

Any rest request send for rule processing will match above rule JSON and trigger api call for all matching entries. The url, retry-count & timeout can be configured under the rule. Using re-trigger, the chained dependent api calls can be triggered i.e. response of API 1 can be send as payload to API 2. Note that all headers sent in request are forwarded as it is.

curl -X POST ${upliftServerUrl} -H 'Content-Type: application/json' -d '{"key":"value"}'

b) Direct Failsafe Api Processing : Below is an example where we don’t need to match a rule from configuration but only need failsafe guarantees. In this case, we are explicitly passing all params. Below is a sample curl.

curl -X PUT ${upliftServerUrl} \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "${url}",
  "payload": {
    "key": "value"
  },
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded"
  },
  "method": POST,
  "maxRetry": 2,
  "timeoutInSec": 30,
}'

Applicability Constraints :

a) Idempotent APIs : As we are going to retry in case of failures, we need to ensure that those apis called by this system should be idempotent. For example, if we’re creating a new payment entry, this shouldn’t create a duplication. This can be handled via a unique idempotent-key.

b) Asynchronous or Event Based APIs : We can utilise these capabilities mostly in an asynchronous or event driven system. For example, if we have a system that requires us to fetch some user details from a downstream service and take action on that data immediately; any retries would ideally be suited via in memory calls only.

Performance Under Load :

We can attest of having an optimal performance with invocations of around 30 rpm with execution time under ~500ms & memory utilization of about 120 MB/call.

Above metrics assured us of optimal success and we have been able to utilize the same further in many of our idempotent asynchronous processing.

Why Uplift : This has helped us in making our inter service communications more stable by providing a failsafe and tolerance mechanism for asynchronous and event based APIs. By providing failsafe guarantees of api success, this helps a lot in case of outages or high load over the system.

Uplift is just one of the technical innovations being done here at NoBroker. If you’re excited to read more about these challenges and the solutions created here, come join us !!