Inside Daily Fire — Receiving a Slack Event
For the last two weeks I’ve been hard at work re-structuring how the backend is architected. I was greatly influenced by Domain Driven Design (DDD) principles and blog posts about how others have used DDD in Rails apps. In this post I want to dive into a section of the code base and show you how Slack events are handled by the backend. I hope that this post will be both a place where you can give me some feedback and pointers, but also educational to those wanting to learn more about software and design patterns.
First a bit of context
DDD encourages you to identify a ‘Core Domain’, and break your app up into seperate ‘Domains’ . Domains are a pretty high level concept: “A sphere of knowledge (ontology), influence, or activity” 🤔 ⁉️ [Wikipedia] so it might be easier to explain with examples.
In Daily Fire’s case I identified the core domain to be the code that was related to Users, Groups, Songs and the associations between them (i.e. a user has many songs, a user is a member of a group).
I then identified other Domains in the codebase. Inspired by the post Modular Monoliths I took to extracting these Domains into their own Rails engines. I found that they mostly formed around the integrations that Daily Fire has. For the moment, they are Slack and Spotify. I also extracted an API Domain, containing all the API logic.
I won’t go into too much detail on the setup of these in this post, but if that is something that you would like to learn more about please let me know!
The prefix df- means that the code in that folder contains Domain Logic (code and concepts specific to Daily Fire)
In this post I want to take a closer look at the Slack Domain (referred to as df-slack). More specifically, what happens when a new message is received from Slack.
Receiving an event from Slack
Daily Fire uses the Slack Event API to receive messages from Slack. Unfortunately, Slack does not allow you to subscribe to just a single channel, and instead sends you every message from every public channel. This means there are a lot of messages sent to the Daily Fire Slack app endpoint which Daily Fire doesn’t care about. It also means, as more teams add Daily Fire, the number of messages Daily Fire receives increases exponentially (or something like that 😕). To mitigate this, I set up a AWS Lambda function that filters all the messages and only forwards message events that are from a Daily Fire connected channel. The benefit of using a Lambda function here is that it is really cheap, fast, and scales automatically.
Now that we are receiving only Slack events from Daily Fire channels we can start processing them and trying to get some meaningful information from them.
Lets break this down,
Here we assert the event has a valid structure (i.e. checking that the event has actually come from Slack).
On line 22, we then check if the message contains links. To make this easier I extracted a gem for common parsing tasks that are required throughout the different Domains of Daily Fire. This message_parser simply takes a message and exposes two methods: .contains_links? and .links. These methods allow me to extract the links out of the message easily.
We then check if there are valid links.
Wait… Didn’t we just do this?
Kind of..
Daily Fire currently only supports a few types of links: Spotify, SoundCloud and YouTube. To determine whether the links are valid, we take the links returned from the message_parser and check them against a few link_matchers. These are wrapped in another type of “parser” called a link_parser, which we can use to ask which service a given link belongs to, if any.
On line 24 and 25, we check if this event is a duplicate event or a duplicate link. Slack tends to send you more events than actually occurred, so here we check if the event has already been processed and saved, dropping it if it has.
Now that all the validation has been done, we save the Slack event using the repository pattern. By saving the event to the database we can refer back to it in other parts of the codebase, like for responding to the Slack message with the Bot!
Line 34 introduces a DF::Common::PubSub::Publisher class. This is important, and is how different domains in Daily Fire talk to each other. DDD encourages you to keep different application concerns seperate. Here what we are trying to do is notify the df-core domain that a new post has been received. However reaching into df-core from df-slack would be against the principles of DDD. This Publisher class takes care of that for us, I won’t go into detail about it here, but all you really need to know is that listeners can subscribe to events they would like to be notified of, in this case: new_post_received.
The other important thing here to note is the contents of the event body
Because Daily Fire integrates with external chat services I extracted the concept of a remote_account and a remote_source. How these are related to Users and Groups is a df-core concern, and not for df-slack to deal with.
An interesting characteristic of a remote_account and remote_source is that they are identified by a deterministic UUID. A UUID is a universally unique identifier, and deterministic means that ‘a given input will always produce the same output’. This UUID is generated based on the provider, in this case slack, and the remote_user_id or remote_channel_id. The generation logic of the UUID’s is in a shared gem called df-uuid_generator.
This means that Domains that deal with remote accounts, like df-slack, can easily generate a UUID that can be shared throughout the system. Then df-core only needs to deal with a UUID and doesn’t have to know where the remote account came from or how it was generated.
This provides a nice extensible way to deal with remote accounts. For example, when Daily Fire starts handling Facebook Messenger accounts it doesn’t need to be updated, as they will simply be identified by a deterministic UUID too. I’ve included a diagram to help illustrate this concept:
The other notable attribute of this event body is the actual remote_account, remote_source, and remote_event objects. These are Data Transfer Objects (DTO) which is a rather fancy way of describing a really simple idea. These are just immutable objects that make it easy to pass around data in object form. By using a defined DTO it also makes it easier for consumers to reason about the attributes of the object as they are documented in code:
Now that we understand the event body we can move onto the consumer of the event, df-core
Turning a post into a song
The new_post_received event arrives at a similar looking class called ProcessNewPost in df-core. As the name would suggest, this class processes a new post. I think it is important to note here that a post and slack event are now two different concepts in the codebase. This means that when adding new ways to post to Daily Fire they have a clear interface to go through.
Here we can see that the arguments UseCase takes are the same as the event body passed from df-slack.
Here in the .process method we are looping over all the links in the post (L25), and saving a Song for each of them (L29).
This Song object will be populated with the User and Group relationships (L31,32), along with a reference to the remote_account and remote_source via their UUID’s (L34,35). The important part here is that it is clear that User, Group and Song are all df-core concepts.
We then publish a new event: new_post_saved
Notice here that we are using dto’s again — a nice, expressive way to share data between domains.
On line 53, we see that there is another event emitted from this class: new_remote_account_detected. This is emitted if the post is from a remote account Daily Fire hasn’t seen before. Subscribers of that event can then fetch more information about it. For example, df-slack can then react to every new remote account from Slack, fetching some information and sending it back to df-core.
Wrapping up
Well.. If you made it this far, thanks for reading :-) Now you know what happens every time you post a link to a Daily Fire Slack channel.
I hope that was interesting and you learnt something. I’d also love to get some feedback on this approach, if someone has done something similar or if you see anything I could improve upon, i’d love to hear it! This my first attempt at doing anything with Domain Driven Design, and although it takes a fair bit more thought and effort (I did strict TDD while doing this too) I am really happy with the results. I feel like the codebase is in a better place than it was at the start of the two weeks, and is in a better position to be extended upon from here. It has provided a lot more clarity over the different parts of the system and their responsibilities.
I really like the Rails Engine approach as it gives you a nice way to seperate parts of your application while being able to plug them into your main app. It also makes it really clear when you are trying to reach into other domains for something, encouraging you to find a better way.
There are many parts to the Daily Fire system. If this post is something you enjoyed please let me know with a clap or two, and perhaps even a comment about a specific part you would like to see. You may be interested in the Vue.js front-end, the overall architecture of the backend, how songs are fetched, I don’t know, I’d love your input! Do you want to see more code? Was this too technical? Want to learn about the frontend?
Thanks for reading, and have a great day!
Peace, Love & 🔥
