Don’t call us, we’ll call you: Sending webhooks with Rails

Benedikt Deicke
Feb 9, 2018 · 9 min read

Integrating with other services is on the roadmap of almost any SaaS application. There’s only so much your application can do itself. Sometimes it’s best to leave tasks up to others who specialize on them.

Building specialized integrations with other tools is one way to approach this. However, wouldn’t it be nice if your application had a generic way for others to be notified about things that happen on your side so they can work with that information?

When it comes to passing data around in (near) realtime, webhooks are the way to go. There is no real standard for them, but most services are working with simple POST requests that send JSON as their request body. Webhooks are triggered for different events and transmit relevant data to other parties.

In the following, we’ll be talking about how to allow other services to integrate with your application, by allowing them to register webhook endpoints and notifying them when something interesting happens.

Registering webhook endpoints

In order to send webhooks, you have to know where to send them to. You have to provide your users with a way to set up new webhook endpoints. This can be done via user interface. In it’s simplest form, it would just be a textfield where users can enter a URL. In more advanced scenarios, you might allow them to specify multiple different endpoints and the types of events that should send a webhook to these endpoints.

To make things easier for your users, you can also decide to allow webhook endpoint registration via your API. This way, other services can just setup the endpoints they need, making things a lot easier for your users. Add authentication via OAuth on top, and it can be as easy as a single click of a button.

We start off by implementing a Webhook::Endpoint model. Each endpoint needs a target url, a set of events that it cares about, and an account this endpoint belongs to. To make things a little easier for ourselves, we use PostgreSQL’s array columns to store the events as a simple list.

class CreateWebhookEndpoints < ActiveRecord::Migration[5.1]
def change
create_table :webhook_endpoints do |t|
t.string :target_url, null: false
t.string :events, null: false, array: true
t.references :account, foreign_key: true, null: false
t.timestamps
t.index :events
end
end
end

This example uses namespaces to keep things a bit more organized. We put all webhooks related code into a webhook directory. In the database we prefix all webhook related tables with webhook_.

# app/models/webhook/endpoint.rb
module Webhook
class Endpoint < ApplicationRecord
def self.table_name_prefix
'webhook_'
end

# ...
end

To make the events array column work, we have to tell Rails about it by adding an attribute macro to the model.

attribute :events, :string, array: true, default: []

On the validation side, we only want valid http urls as target url. Luckily Ruby comes with a handy URI regular expression in it’s standard library, ready for us to use. We also add a presence validation for the events, so every endpoint is triggered by at least one event.

validates :target_url,
presence: true,
format: URI.regexp(%w(http https))

validates :events,
presence: true

In order to quickly retrieve endpoints for one (or more) events later on, we implement a custom scope. It uses PostgreSQL’s contains operator (@>) to just return the endpoints interested in a particular event.

def self.for_event(events)
where('events @> ARRAY[?]::varchar[]', Array(events))
end

Notice that we’re using Array(events) to convert the methods arguments into an array, or just leave it untouched if it already is one. This allows us to call the scope with one event name, or an array of multiple ones.

Next, we add some normalization and sanitization logic to the events setter method. This can be done, by simply overwriting the method, doing the necessary changes, and then calling super.

def events=(events)
events = Array(events).map { |event| event.to_s.underscore }
super(Webhook::Event::EVENT_TYPES & events)
end

In this case we’re normalizing the events into underscore strings. Afterwards we intersect the resulting array with a predefined array of event names (more on that later), to make sure all the events name are actually valid.

Finally, we add a method to deliver an event to this endpoint. It’s empty on purpose for now.

def deliver(event)
end

With the Webhook::Endpoint model in place, we need a controller, the view layer and most likely an API representation for it. There’s nothing special about the implementation of all of this, so we won’t cover it here.

Representing events

As mentioned before there is no standard on how webhook payloads should look like. In this example we use them to represent events that are triggered within your application. Assuming the application somehow works with a Project model, here’s an example of an event for a newly created project.

{
"event_name": "project_created",
"project": {
"name": "Example Project",
}
}

The webhook payload follows a simple structure. It always has an event_name and a set of embedded models with relevant data. You’re of course free to settle on a different format. Instead of embedding the event name in the payload itself, you might choose to send an X-Event-Name header with your request. Whatever you do, just stick to a pattern for all the webhooks you’re sending and document it. All the developers building integrations with your application will thank you for it.

Let’s implement a Webhook::Event model to represent a single event. This can be done as “plain old ruby object”. If you want to keep track of the state of every single event, you can store all events in the database as well. For the purpose of this example, we’ll keep it simple.

The implementation of Webhook::Event is straight-forward. It’s an object with two instance variables. One for the event’s name, and another one for its (optional) payload. To keep track of valid events, we add the EVENT_TYPES constant used earlier. Whenever you add a new event, add it to this list.

# app/models/webhook/event.rb
module Webhook
class Event
EVENT_TYPES = %w(
project_created
project_updated
project_destroyed
)
.freeze

attr_reader :event_name, :payload

def initialize(event_name, payload = {})
@event_name = event_name
@payload = payload
end

# ...
end

As we want events to be delivered as JSON in the HTTP request’s body, we have to implement serialization for this event model. This can be as simple as taking a copy of the payload and adding the event_name to it.

def as_json(*args)
hash = payload.dup || {}
hash[:event_name] = event_name
hash
end

However, we want to keep our code clean and not come up with the entire payload for every single event. If you’re using ActiveModel::Serializers to handle serialization of your API resources, you can leverage it to properly serialize the payload for you.

def as_json(*args)
hash = payload.transform_values do |value|
serialize_resource(value).as_json(*args)
end

hash[:event_name] = event_name
hash
end

private

def serialize_resource(resource)
ActiveModelSerializers::SerializableResource.new(resource, {})
end

This way, you can create events and simply pass your model instances as payload.

Webhook::Event.new(:project_created, { project: project })

Triggering events

With both Webhook::Endpoint and Webhook::Event in place, it’s time to actually trigger events. To keep things organized and DRY, we use a module to implement this. Whenever you want to trigger events, you just include that module in your model and call one of the helper methods.

We start off by adding a webhook_scope method. It’s just a stub, that has to be overwritten by each model to return the scope of the webhooks. If you’re getting a bit confused by that, don’t worry. It’ll get clear once you see the module used in an actual example in a few moments.

# app/models/webhook/delivery.rb
module Webhook
module Delivery
extend ActiveSupport::Concern

def webhook_scope
raise NotImplementedError
end

# ...
end

Next, we add a method to trigger events. It just takes two arguments. One for the event name, and one for its payload. It then constructs an Webhook::Event object from those and asks every endpoint interested in the event to deliver it.

def deliver_webhook_event(event_name, payload)
event = Webhook::Event.new(event_name, payload || {})
webhook_scope.webhook_endpoints.for_event(event_name).each do |endpoint|
endpoint.deliver(event)
end
end

To make things even simpler, we implement a default webhook_payload method, as well as a deliver_webhook method. The latter relies on a naming convention to generate an event for the current model based on an action name.

def webhook_payload
{}
end

def deliver_webhook(action)
event_name = "#{self.class.name.underscore}_#{action}"
deliver_webhook_event(event_name, webhook_payload)
end

Coming back to our projects example from earlier, we can now trigger events like this:

# app/models/project.rb
class Project < ApplicationRecord
include Webhook::Delivery

after_commit on: :create do
deliver_webhook(:created)
end

belongs_to :account

private

def webhook_scope
account
end

def webhook_payload
{ project: self }
end
end

As most events revolve around creating, updating, and removing models in your database, we can also build a simple module to handle those events automatically.

# app/models/webhook/observable.rb
module Webhook
module Observable
extend ActiveSupport::Concern
include Webhook::Delivery

included do
after_commit on: :create do
deliver_webhook(:created)
end

after_commit on: :update do
deliver_webhook(:updated)
end

after_commit on: :destroy do
deliver_webhook(:destroyed)
end
end
end
end

Please notice that the module is using after_commit callbacks. It’s important to only send webhooks after the data is actually present in your database. This is to prevent weird “not found” errors in the occasions where webhooks are delivered faster than your database takes to commit the transactions or it’s part of a larger transaction that fails shortly afterwards.

Using the Webhook::Observable module, adding webhook events to a model becomes as simple as this:

class Project < ApplicationRecord
include Webhook::Observable

belongs_to :account

private

def webhook_scope
account
end

def webhook_payload
{ project: self }
end
end

Delivering webhooks

To deliver webhooks, I strongly recommend using a background job queue. As delivery depends on 3rd party services, you should expect them to be slow or even unreachable. As your application might send webhooks to multiple endpoints per event, you definitely don’t want your users having to wait until all of them are delivered. Using a queue also allows for retries when delivery fails for some reason.

In this example, we’re using Sidekiq as background job queue. This is not a requirement, though. ActiveJob with any backend you like will to this job as well.

# app/workers/webhook/delivery_worker.rb
require 'net/http'

module Webhook
class DeliveryWorker
include Sidekiq::Worker

def perform(endpoint_id, payload)
return unless endpoint = Webhook::Endpoint.find(endpoint_id)
response = request(endpoint.target_url, payload)

case response.code
when 410
endpoint.destroy
when 400..599
raise response.to_s
end
end

# ...
end

The implementation will raise an exception for all HTTP status codes that indicate some sort of error. Sidekiq will catch the exception and just retry the delivery a few moments later. Please know that HTTP status code 410 Gone is an exception. It’s a small idea picked up from Zapier. Whenever the endpoint returns a 410, we consider it no longer valid and just remove it from the database.

The actual HTTP request is nothing fancy. As a result, we just use Net::HTTP from Ruby’s standard library.

def request(endpoint, payload)
uri = URI.parse(endpoint)

request = Net::HTTP::Post.new(uri.request_uri)
request['Content-Type'] = 'application/json'
request.body = payload

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = (uri.scheme == 'https')

http.request(request)
end

Finally, we add the implementation of Webhook::Endpoint#deliver we skipped earlier. It enqueues a new Webhook::DeliveryWorker for the given event.

def deliver(event)
Webhook::DeliveryWorker.perform_async(id, event.to_json)
end

It’s important to serialize the event model in the very moment the delivery is triggered. Otherwise the payload might have changed until the delivery takes place, resulting in weird data and confused developers.

Taking things one step further

While the implementation shown here covers the most important parts to get webhooks implemented in your application, there are a few more things you should consider.

Allowing for authenticity checks

As webhooks are usually delivered to more or less public endpoints it’s best to add some way that allows the receiving application to ensure the data is actually coming from you. One common way to do this is to add a signature somewhere in the payload. This can either be done in the JSON body or in a custom HTTP header. The signature should be based on the payload itself and a secret only known to the sending and the receiving application.

Accidental Denial of Service attacks

Based on the details of your application, it might happen that you send a lot of requests to the endpoints. Some of them might not be able to handle the load. With retries in place, the number of requests sent to the endpoint will only increase, making things worse. It’s a good idea to have some sort of rate limiting in place or even temporarily disable an endpoint that is unable to process requests right now.

Wrapping up

Webhooks are a great way to allow applications to integrate with yours. At their core, they’re simple and easy to understand HTTP requests. With just a little bit of code, they allow your users to use your application in ways that you didn’t think of before.

You can get the source code of everything explained in this article at GitHub’s Gist. You should be able to use it in your existing applications, with just minor modifications.


This article is an excerpt from The SaaS Guidebook. It’s just one of the topics the book will cover. If you’re interested in building and running solid SaaS applications, please sign up for updates on the book.


Originally published at benediktdeicke.com.

Benedikt Deicke

Written by

Freelance Software Engineer (Ruby & JavaScript). Co-founder at @Userlistio. Creator of @StageCMS. Loves music, food, and cooking.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade