Event-Based Communication With Zendesk

How we used Amazon EventBridge to reduce our reliance on API-based integrations

If you’ve designed a service that is reliant on a third-party API then you may be susceptible to rate limit impositions. This can cause headaches for engineers because failure scenarios usually result in unhappy people. 😒

Simply monitoring and reducing API usage might remedy the situation, but the best resolution is to try and decouple your system from the API.

Let’s delve into how we recently started using Amazon EventBridge to achieve this and make our communications with Zendesk more asynchronous.

How and when Just Eat uses Zendesk

We leverage Zendesk’s Ticketing System to gather all of our support requests. They’re typically generated from a variety of sources/integrations, so being able to manage them in a single system allows us to prioritise and process requests while being able monitor the activity of our support agents.

As the number of users on our platform increases, we have seen an increase in the number of support requests. We design workflows to help our agents manage this, which often involves us having to manage support tickets programmatically.

There are a multitude of scenarios which might require us to create and manage a form of support ticket — some examples include:

• A customer raising an issue with an order, via chat or webform
• A delivery partner requesting support regarding logistics

These scenarios manifest in the creation of a ticket within Zendesk, which is managed until resolution. One of our support agents may be required to manually intervene at various stages, but we automate as many steps as we can to reduce the handling time of each ticket.

Since Zendesk is our primary means of handling support requests, it also becomes an important data source for reporting. Gathering statistics from Zendesk Support gives us valuable insight into our effectiveness in handling customer queries. Obtaining these stats often involves extracting large amounts of data from Zendesk into a local data store.

Current problems

Most of our programmatic interactions with Zendesk are done using their API. Due to our volume of support requests, we end up making heavy usage of this API to create, update and retrieve ticket data.

documentation for Zendesk’s API: developer.zendesk.com/api-reference/

Zendesk imposes limits on API usage, which we have exhausted on numerous occasions at peak times. When this happens a number of our applications will start seeing challenges when supporting the workflows that rely on Zendesk. Often this will mean we need to manually intervene to support the workflow. The result then impacts the quality of service we are able to provide to our various end users.

Finding a solution

We needed to find a way of altering our applications in order to reduce our API usage.

Quick wins

It should be noted that one simple resolution might be to increase API usage allowance, by throwing money at the problem. At time of writing, the imposed API limits depend on your plan type with Zendesk Support. If you are on an Enterprise plan the limit is 700 requests per minute. However, it is possible to acquire a High Volume Add-On which increases a plan’s limit to 2500 requests per minute.

Of course, this approach does not actually do anything to reduce reliance on the API, but it can work if you can be sure that your usage is unlikely to increase anytime soon. For us, this approach was not a long-term solution.

up-to-date Zendesk documentation for API usage: developer.zendesk.com/api-reference/introduction/rate-limits/#zendesk-support-plan-limits

Tweaks to current approach

We have adopted a few different approaches in an attempt to manage API usage at peak times. For example, we have tooling that generates reports based on support requests by extracting data from Zendesk at regular intervals. This extraction process resulted in many GET requests to Zendesk. We made a couple of alterations here:

• Run this extraction process outside of peak hours
• Where possible, we used the Incremental Export endpoints. This is different to just getting one or many tickets because it allows you to retrieve tickets that changed or were created in Zendesk since the last request

A shift in approach — moving away from our “pull” architecture

For reporting applications, tweaking when and how we made certain HTTP requests worked well, and wasn’t really a big shift from what we already had. However, we still had applications which needed to be more reactive to data changes in Zendesk.

For example, we wanted to trigger a business process the moment an agent solved a support ticket. For this, we needed one of our applications to respond to a support ticket status change. The approach we took for our reporting applications does not work here. We can’t respond out of hours, and exporting large amounts of ticket data periodically doesn’t actually help us identify a status change.

Our initial approach was to poll the ticket audit endpoint periodically with a relatively high frequency. This endpoint is useful because each audit represents a single update to a ticket. We could then scan the responses for any events that signified a ticket had been solved. Doing this effectively meant we were monitoring all ticket changes in Zendesk, at discrete time intervals. Whilst this approach worked, it had some issues:

• We were processing a lot of audit events we did not care about. We only cared when the status ticket of a ticket changed — not when a comment was added, for example
• We weren’t reacting to state changes when they happened. We only found out about changes every time we asked for them, which we did not want to do frequently to avoid exhausting our API usage allowance
• It required a fair amount of code to test and maintain

Leveraging AWS Event Bridge

Amazon EventBridge is a serverless event bus which allows you to stream data from a number of supported third party SaaS applications into your own AWS account. One of the supported partner integrations is Zendesk. This means we can stream events from a single Zendesk instance into our own AWS account and route them to AWS services, such as AWS Lambda and Amazon SNS.

The full list of supported partner integrations can be found at https://aws.amazon.com/eventbridge/integrations/

Currently, Zendesk only produces events relating to its ticketing system. You can find the full list of available events here.

Going back to our use case, in order to react to the status change of a ticket, we can make use of their Status Changed event. By subscribing to this event, we can trigger the business process that was previously reliant on our polling approach.

An example event will look something like:

{
  "version": "0",
  "id": "5cabfa42-9882-4f86-9a13-be7097e63171",
  "detail-type": "Support Ticket: Status Changed",
  "source": "aws.partner/zendesk.com/9242270/default",
  "account": "123456789012",
  "time": "2019-05-25T01:23:48Z",
  "region": "us-east-1",
  "resources": ["Support Ticket"],
  "detail": {
    "ticket_event": {
      "meta": {
        "version": "1.0",
        "occurred_at": "2019-05-25T01:23:45.721021468Z",
        "ref": "1-1234567890",
        "sequence": {
          "id": "3A0D9570D25C80909594821C606343B5",
          "position": 1,
          "total": 1
        },
        "actor_id": 20978392
      },
      "type": "Status Changed",
      "previous": "open",
      "current": "solved",
      "ticket": {
        "id": 35436,
        "created_at": "2019-05-20T22:55:29.721021468Z",
        "updated_at": "2019-05-25T01:23:45.789534301Z",
        "type": "question",
        "priority": "low",
        "status": "solved",
        "requester_id": 20978392,
        "submitter_id": 76872,
        "assignee_id": 235323,
        "organization_id": 10002,
        "group_id": 98738,
        "brand_id": 123,
        "form_id": 6876543,
        "external_id": "TEST1234",
        "tags": [
          "enterprise"
        ],
        "via": {
          "channel": "web"
        }
      }
    }
  }
}

At a high level, the steps we took in order to process these events were:

1. Using Amazon EventBridge, create an event bus³ that receives events from Zendesk²
2. Configure this event bus to route⁴ Status Change events to an SNS topic⁵
3. Pull events off the SNS topic to react to status changes via a Lambda trigger⁶

Let’s have a more detailed look at exactly how we achieved these steps:

1. Set up the Zendesk Events Connector

Basically linking a single Zendesk instance to one of our AWS accounts. Instructions on exactly how to do this has been documented well by Zendesk, here.

An AWS account is a requirement for creating the link between the Zendesk Events Connector and Amazon EventBridge.

Within the Zendesk Admin Center, we set our desired AWS account as a destination for the events produced from that Zendesk instance. Doing this created a new event bus in our chosen AWS account.

2. Configure the event bus to route events to an SNS topic

We now have an event bus associated with the Zendesk event source. From here, we added EventBridge rules (associated to that bus) that will match on incoming events and send them to targets for further processing.

Using event patterns, we can configure a rule to match only on Status Changed events, so we need not process any other type of ticket update. You can see from the example JSON above that these events have a detail-type value of Support Ticket:Status Changed. Event patterns have the same JSON structure as the events they match on, more detailed information can be found here. Our rule is relatively simple — we want to match on events whose detail-type is Support Ticket:Status Changed, so it looks something like:

{ "detail-type": ["Support Ticket:Status Changed"] }

When creating the rule, we configured the target to be an SNS topic of our choosing.

3. Subscribe to events from the SNS topic

That brings an end to the “EventBridge stuff” — there is an event bus routing events to an SNS topic. Now it is just a case of pulling these events off the topic. We chose to set up a lambda trigger, which is effectively called whenever Status Changed events from Zendesk are produced.

Conclusion

This process of subscribing to Zendesk events via Amazon EventBridge has reduced our dependencies and overhead from API based integrations with Zendesk.

We discussed one scenario, in which we obtained data about status changes, but we can take this approach anytime we want to obtain ticketing data. Some other use cases I can think of:

• Calculate the number of tickets created per day: subscribe to the Ticket Created event and increment some sort of counter until the end of each day.
• Keep a local data store of ticket information: let’s say you frequently query Zendesk for ticket data but only care about the ticket ID and description. You could subscribe to Ticket Created and Description Changed and store the data in a local database. This means your frequent queries would use your database, rather than Zendesk’s API.

At Just Eat Takeaway we strive to make services communicate using events. Historically, our services have been making direct service-to-service requests with Zendesk, and Amazon EventBridge helped us move away from this model in favour of an event-driven approach.

Just Eat Takeaway.com is hiring! Want to come work with us? Apply today.