Slack App Testing
You down with testing in OPP?
Back in April we released our automated digital marketing assistant in the Slack App Directory. Slack is awesome and the app ecosystem is only getting stronger (especially considering Slack just announced a massive funding round). But as much as we love being a Slack app, one of the biggest challenges we encountered was around end-to-end testing. In this post I’m going to explain why we decided to use Cypress.io. I’m also going to show you some examples on how we are using it.
So why is end-to-end testing difficult in Slack? Well, our product lives and breathes behind an interface that another company built and manages. Slack has built a huge ecosystem to allow developers to piggyback on their existing platform and user base. Developers have built thousands of Slack apps that are easy to find in the Slack App Directory. Slack is not unique in this regard. There are a lot of products/platforms out there that allow developers to deliver value by building within existing platforms.
Another great example of this trend is Intercom. They recently opened their App-Store to developers and have hundreds of apps. Here’s a great example from our buddies at Userfeed.io. Landon Bennett and Kyle Conarro have built a product that integrates with Intercom’s ecosystem to capture user feedback.
Both Slack and Intercom are part of a larger trend and are what Mary Meeker calls “messaging threads” in her 2018 Internet Trends report.
This presents numerous challenges for developers and Cypress.io is helping with what we like to call, testing within Other People’s Products. You down with OPP?
OPP
As I mentioned at the beginning, when building within OPP you have trade-offs, some that are beneficial and others that are challenging. Here are the things I focus on with this post:
Benefits
- Initial UX is typically established
- Most platforms have marketplaces or app stores where users can find your app
- No UI to build
Challenges
- You are limited to UX of the platform
- You have less control over user base
- No UI to run end-to-end testing
As a developer that lacks those “make it pop” creative skills, one benefit about building tools in OPP is you don’t build the UI. In our case, Slack handles that and they have done a great job. What we do need to focus on though is the UX or User Experience. Most products have features or UI elements you can use like Slack’s dialogs, buttons, and menus. Those features limit your experience with UI and constrain your UX so it does take a level of skill and gusto to get it right.
Another great OPP benefit is these platforms have easy ways for people to find your product with app directories, marketplaces, or app stores. Since these market places are highly public, you do end up having less control over which users try your product. Any user of your parent product, in our case Slack, could potentially be a user of your product. Messaging and user experience become very important to ensure you are on-boarding the right people.
These first couple challenges can typically be managed with meetings, customer feedback, and trial and error. The biggest challenge we faced when building for OPP was testing, specifically end-to-end testing.
Testing Our Slack App
We use AWS’s API Gateway and Lambda products. These are great and easy to use once you wrap your head around them. I’ll have another post around how we are utilizing the Serverless framework to easily manage, build, and deploy to AWS. Due to the serverless nature and decoupled architecture using Lambda, we can easily write unit tests to ensure each function is doing its job and have our build pipeline setup using AWS CodeBuild. We have thousands of tests running every day, all day.
Most applications developed these days use third party integrations and APIs. The majority of systems and data we collect are from other platforms as well (e.g., Facebook Ads API and Google Ads API). Most of this work can be tested and stubbed/mocked with common testing frameworks. We are using Node.js and have recently embraced Jest but have some older stuff using Mocha.
In the example above, we are constructing our expected message that we should send to Slack. On lines 3 and 4 we are actually mocking (creating a fake) Slack client so that our testing pipeline is not calling their API during the thousands of unit tests we run all day. Unit testing covered!
We had a larger problem. Since we are completely integrated into a separate stand alone product, how can we run end-to-end testing to make sure everything works as expected? How the heck are we going to test our Slack app in Slack?
Enter Cypress.io
Our ATDC friends down the hall at Cypress.io have created an amazing product to help us do just that…test our Slack app in Slack. Since the main Slack app can run in a web browser, Cypress can handle our testing needs. Many of our features require multiple Lambda functions, back and forth user interaction and functionality that can only be tested through Slack. The ability to easily run our end-to-end tests was reason enough to use Cypress, but here are a few features that we really appreciate:
- Easy to setup and easy to use. You add the cypress module to your package.json and you are ready to start testing.
- Network request management allowing better control and detection of request and response data.
- Screenshots for test runs, showing the test result information and the browser view at the time of the test.
- Videos of the entire test run. We have actually used this to discover other bugs while investigating a particular test failure.
Easy Setup and Use
Cypress has an easy setup. You can follow the simple install process here. Once its added and downloaded, you can start writing tests. I’ve used a lot of tools in my career and it’s refreshing when documentation and examples are done well. Here is a link to their example repository, what they call the kitchen sink. It has examples detailing all aspects of the tool.
Due to the simple install process and test writing, we were able to have one of our junior developers build out our tests using just the docs and examples. I don’t think that would have been the case with other products. I’ve used another tools before and to get them up and running was a challenge even for seasoned engineers.
Thankfully Slack uses Cypress best practices for making element selection easier. Most objects have a data-qa attribute. Selecting elements is less brittle because those attributes do not change as much as classes and ids might. They even have a suggested selector tool that gives you the selector data for an element just by clicking the ui. In the screenshot below you can see the selector that was created from clicking #general channel on the left.
Network Requests
They built their product from the ground up to completely manage the browser. This gives the tool direct access to data coming in and out of your application. Network request management is possible because they handle the entire network activity natively instead of by use of bindings and libraries that need to be installed. Since Cypress is managing that data directly, it automatically waits and gives visibility and control over that information. Waiting for network requests to complete is simple, no need for sleep timers or random waits like with other tools. Below is an example of how we wait for Slack’s dialogs to appear.
You can even record and reply network traffic to ensure the response is correct based on the request using fixtures. Currently we’re testing that data is being sent to Slack and the messages are presented correctly so we have not used fixtures yet. We are going to investigate fixtures for our nightly alerting which is time sensitive. Being able to guarantee certain sets of data is going to make fixtures very useful.
Writing tests with Cypress is similar in nature to how we are doing our unit tests with Jest or Mocha with describe and it blocks.
In the example above, we are looking for the input box to type our welcome slash command. Using the route network request functionality, Cypress automatically waits for the specified request to complete. We then check for the output message and ensure it contains the right information and a settings button.
Screenshots and Video
As a developer and ex-QA engineer, the screenshot and video capture features could not be more important to me. Many times it’s hard for the QA team to explain what’s happening when a test breaks. Cypress makes this a breeze. Here is an example test of our settings dialog. In the screenshot below, the Company Profile dialog expected the website url to be empty but it was populated with Matt.com.
Cypress goes one step further and actually provides the video stream of the test run! A developer can see exactly what’s happening as the test happens. Here is an example video below of what appears to be a very simple feature of our app, adding an ad account. Behind the scenes, not so much! There are a number of Lambda functions in place to ensure proper interaction with the user, proper interaction with Facebook or Google, and then making sure the user is notified accordingly. In the video, the test fails at the end because we updated messaging for the Google AdWords to Google Ads update.
What’s Next
We are just scratching the surface with what we can do with Cypress. We still want to check out fixtures for our next round of testing. They have a great dashboard to see status of runs, watch replay videos, and much more.
You can also run the entire test suite headless. This allows you to push your tests into your CI/CD pipeline. Now that we have our initial test suite up and running, we will be working on that. Cypress provides docker images and examples for helping us get that working. As I mentioned, we are using AWS CodeBuild which is not one of the recipes so we are in uncharted territory...let me know if anyone else has set that up.
Thank you Cypress.io for give us such a great tool. We have enjoyed using it and can’t wait to see what you do next.
Building for other people’s products? Let us know how you get down with OPP!
Later,
If you are interesting in seeing what we have built here at Eletype, you can check our app here in the Slack App Directory and subscribe to our product launch on Product Hunt here.
