Scaling Slack Events in Node
Hacking / Learning / Sharing
If we have some spare time at Founders we love to hack on tools to improve our day to day work. Sometimes these hacks turn into real products that we release to the public.
Besides the value gained from improving our work flows we also gain deeper insight into platforms like Slack. In-depth knowledge of a platform can be a great source of inspiration for new hacks and features. To expedite further development we start to build platform-specific tooling for use in both Founders and portfolio companies. In this post we are sharing a way to scale the handling of Slack events in node.js. There is a small bonus tool at the bottom.
Getting started with Slack development
When we started working on Lurchr we took advantage of Botkit — an all-in-one node.js toolkit for Slack. Botkit allows you to hit the ground running and start interacting with your Slack team in minutes. This is great way to flesh out new ideas and experiment with the platform.
After launching Lurchr on Product Hunt we quickly got to the point of connecting to hundreds of different Slack teams and started to hit some of the limitations of Botkit. Each team your bot connects to you have to spawn a new Botkit instance. This means you for each team have:
- WebSocket connection (Slack by default delivers events and messages to your bot over WebSockets)
- Event Listener (allowing you to react to these messages)
- OAuth (for authenticating Slack teams)
- Slack API helpers (
This all takes up precious memory. And if want to scale horizontally you have to shard team.
Events API to the rescue
Luckily Slack also realised that this is not optimal and that their use of WebSockets was a catalyst for this problem. Slack Events API to the rescue! Using the Events API you just have to register a WebHook endpoint with Slack, configure which events you want to receive and Slack will post events from all teams that your application is connected to. This allows you to scale your events service horizontally since it is just a dumb web server receiving JSON and passing it on to whatever needs to handle it.
Update: If you need a way to keep your bot “present” in all the different teams take a look at https://github.com/FoundersAS/slack-rtm-presence
slack-events-listener — Easy to use web server which emits all events received from Slack Events APIgithub.com
To support this pattern we built a small Express middleware slack-events-listener, which you can either use to create a separate events service or enable your existing web server to receive Slack events. The middleware just takes a callback which is called whenever a Slack event is received.
Please note: Slack expects you to reply with a 200 to their request within three seconds. So make sure you aren’t doing anything to heavy before calling cb().
Hopefully this can be useful to others who are exploring the Events API and would love to hear from anyone who uses it. Remember — we accept pull requests :)
PS: If you like hacking on new products and platforms and like OSS — we are always hiring! https://founders.workable.com
PPS: SSL is important!
We have also developed a small and naive middleware that takes some of the pain out of authenticating with Let’s Encrypt.