How we built Steno, your app testing sidekick
A testing tool to record and replay Slack API interactions
This week we introduced Steno, a command line utility that helps you build your app’s test suite by allowing you to record and replay interactions with the Slack API.
We built Steno to address a situation that will be familiar to many Slack app developers. You’ve released a product and the feature requests start rolling in, but you feel a set of mixed emotions – on one side, a feature your users are requesting is powerful and you would love to deliver it. However, you’d have to carefully refactor many scattered pieces of code and you’re worried the whole thing might break.
It’s time to start thinking about tests. You carefully write down each behavior on which your users depend. Now you’re faced with a dilemma: how can you verify that each of these behaviors still work as you change the code? Manual testing requires you to carefully manipulate a team’s state until it triggers each behavior. Sometimes this means creating new users or even new teams. What about automated tests? There’s a flood of information, opinions, and tools, but is there a pattern that’s known to work well for your Slack app?
How Steno works
The Steno testing workfow is followed by some of the highest-quality apps on Slack. Building tests with Steno removes your app’s dependency on our production API from your testing process, so it’s easier to automate.
Your tests will be oriented around scenarios with a known set of inputs and outputs. They are all stored in a readable format, so you’ll be able to test the behavior of your app against changes by re-recording, exchanging, or hand-editing those scenarios. If you’re thinking of using Steno, here’s what else to know:
Steno works with bidirectional HTTP interactions
Slack apps can both send HTTP requests (like when using Web API methods) and respond to HTTP requests (like when using slash commands, interactive messages, or the Events API). Your app’s behavior can involve either, or both, of these styles. Steno records and replays both incoming and outgoing requests, making it simpler for you to write tests.
You can use the language or framework you love
Steno resembles a proxy; in your development and testing environments, you send it the requests you would send to Slack. In record mode, the requests pass through transparently as scenarios are written to disk. In replay mode, you won’t need the internet – Steno will use the scenarios to understand how to respond. From your app’s point of view, the only thing that changes is the hostname in its requests. Steno is outside your app process so you can use any language.
Steno’s scenarios are a readable, portable, and exchangeable format
Scenarios are part of your test suite, so you should check them into source control next to your test code. Scenarios are simply directories in which each file represents a request and response pair, in plain old HTTP, and are so simple you can edit them yourself. You could simplify them by scrubbing out unnecessary or sensitive information, or use them to communicate with others about what came in and out of your app.
Want to know what happens when an upcoming API change is introduced? Start from a previously recorded scenario and mimic those changes. Start Steno in replay mode and you’ll see how your app reacts.
Even sidekicks have an origin story
Building a high-quality software product is like fighting off a masked villain lurking in the shadows. You can’t add quality the way you develop a feature. You look in the dark edge cases to find the defects and eliminate them.
We love talking to Slack developers and learning about their challenges. It’s unsurprising that many developers ask about testing. How should someone test a bot? How can we deal with recreating the messages, members, and events an app depends on? How can we prepare for API changes that aren’t yet visible in our workspaces?
The answers weren’t immediately clear, but the Slack developer community started to share their experiences and even built some interesting tools. We set out to participate in these conversations to amplify the best ideas and to make them accessible to the entire Slack developer community.
Our first idea was to shift as much process as possible from manual to automated. The conventional testing wisdom is to write unit tests for each logical part of your app and mock the interfaces with dependencies, and Slack is a dependency. In practice, Slack’s APIs change as our platform evolves, so the maintenance cost of updating mocks to match Slack’s dynamic APIs is unreasonable.
Our next idea would allow developers to overcome this challenge: instead of programming mocks explicitly, we could build a system to write fixtures from real usage and load those fixtures for tests. The web development community has built many of these systems, and Slack developers who used them reported great results. However, these systems only respond to outgoing HTTP requests, and our platform also uses incoming HTTP requests for events and interactivity. Developers would need to write additional code to mock those requests from our platform. Why couldn’t we have one system to write and load fixtures for both outgoing and incoming requests?
This was the beginning of Steno.We set out to build a tool that would save developers time building and maintaining test cases, which meant it had to be language agnostic, and work at the most fundamental level for all Slack apps: HTTP.
Steno took shape as a command line utility that is packaged and distributed as a single binary for the major operating systems. The project was always for the community, so we knew we wanted to build it as open source software.
Stay tuned for adventures to come
The journey has just begun for your Slack app, for the Slack platform, and also for Steno.
Our first release is a foundation for new ideas to form into new functionality, and we aspire to work with the Slack developer community to build features that will help improve the quality of apps on our platform – and beyond.
Stay updated with the evolution of Steno and star our repo on Github. Become part of Steno’s evolution by sharing your feedback, ideas, and experiences in Github issues.
