Event processing in Rails
Event processing is important because as applications grow, more actions will be taken when something happens, i.e. an event is triggered. Starting out, you might allow users to create content, let’s say a post. Then as you start to think about engagement, you decide to send an email to a user’s followers when they publish a post. Next, as you notice a lot of spammy or inappropriate posts, you want to check the post’s content and possibly flag it for manual inspection. Quickly it becomes the case that you want a single event, a PostCreationEvent here, to trigger a lot of different functionality. In many cases, the functionality is different enough from the core action to move it out of the critical path, but you still want a clean abstraction model for this.
Many companies have sophisticated methods for doing event processing. One common approach is to use Kafka for event logging and to write consumer services that process the data as it becomes available to them. This is nice because you can log a single event somewhere in your code, and consumers will pick up that event when they’re ready to process it. I talked about this a little bit in my last post, event logging in Rails. I think what I missed out on was the fact that setting up something like Kafka takes time and often what developers want, and really need, is a quick but robust solution that offers the same functionality, without sacrificing the abstraction model.
When I think about implementing something like this in Rails, I want it to be clean and out of the way. In the example above, I could imagine a naive approach calling two separate service objects after creating a post — one call for sending an email, and one call for checking content and flagging posts. This to me seems like poor design. What I’d really like to do is just trigger an Event on a post creation and have configuration around event listeners out of my API. This allows anyone to listen (or consume) an event without continuously adding code to the API. All that is really needed here is to add an event consumer and have it listen to the appropriate events.
Working backwards with this problem, we have a few different consumers that need to somehow be called when an event is triggered. Looking to Rails for help we see that ActiveSupport::Notifications is used by Rails itself for a similar problem, logging and instrumenting various events. The next step is to figure out how to tie each consumer to a given event. This is super easy with Notifications and writing the initializer is trivial.
Here we have a POST_CREATION event being mapped to several consumers — one for emails, one for spam, and another for logging to Kafka. The functionality of the consumers is understood, and this nice thing is that they’re logically different and do not need to interact with each other. I do want to note that the above functionality could be abstracted into the consumers themselves, but I prefer explicitness of defining the mapping in this scenario.
Because we want ultimate flexibility with our consumers, we define them to take in a Ruby Hash called a payload. We’ll define a method named `call` on consumers, one that will be called when an event is triggered.
Now, any consumer can do with the payload whatever they want or need to. Writing a new consumer for an event becomes really easy and doesn’t need interaction or coordination with the API controllers at all.
Let’s say that we wanted to add a new consumer to do something when a post is created. Perhaps we want to do a bit of text analysis (a little tf-idf trickery) and determine what the post is about so that we can either add it to a category or surface it in recommendations.
Here we have written a simple consumer that enqueues the analysis for asynchronous execution. While there’s obviously post analysis code missing from here, the only changes we really needed were to write the consumer and update our EVENT_CONSUMER_MAPPING.
The final step is to define what an event is and create an easy way to trigger them. Here, I have defined a base class for such an event, and defined a few methods to help keep things clean.
Now we can finally define our post creation event.
With all of this in place, the last part is to trigger the event, and that can be done with a single line of code.
PostCreationEvent.new(@post).triggerRemoved from the above code snippets, for simplicity and clarity, is default asynchronous behavior, and the ability to opt-in to synchronous processing. Extensive use of this abstraction shouldn’t block or slow down requests, so asynchronous behavior is desirable. But, it might make sense for some consumers to specifically opt-in to synchronous behavior though, so that could be an option, though I might advise against it.
While there are other ways of achieving some of the functionality outlined above (active record callbacks, etc), the point of this is to offer an abstraction — and a damn useful one. ActiveSupport::Notifications takes care of a lot of the heavy lifting and allows us to focus on the important details and the business logic. The ability to trigger arbitrary events and further allow for arbitrary consumers enables application developers to iterate very quickly on new ideas, keep great modularity, and work within a scalable model that allows for future improvements or optimizations when necessary.
