One of the many unique approaches in our attempt to keep at the forefront of software development practices at EventMobi is our lack of a dedicated QA department. This means our lean, agile teams take full ownership of their testing strategies. Without a dedicated QA team, our teams are forced to care about the quality of their tests but shouldn’t necessarily care about the infrastructure behind the testing tools we use.
Hello everyone! I’m Will, a Computer Engineering student from the University of Toronto, and spent over year during my internship at EventMobi.
In order to provide an automated solution that could service a growing team of 25+ developers who are doing multiple deploys per day, we needed to find a scalable solution that would require minimal maintenance.
We identified our testing pyramid, which defines the categories of tests we would run in our CI process. Internally, we had a goal that our test suite should take no longer than 15 minutes (we were previously sitting at roughly 1 hour for the full suite). However, our internal Jenkins build server, used to run our Selenium test suite, could only scale so much. Luckily, we were able to quickly move to Sauce Labs to help us address these needs. Sauce Labs provides us with a scalable, hosted Selenium grid service that requires no maintenance. Individual tests are run on their own fresh VM and can be configured to run on a multitude of different browser and OS platform configurations.
How EventMobi Uses Sauce Labs
Within three weeks, we were able to move from our manually triggered Jenkins build step process to having Sauce Labs hooked into our CI pipeline that is managed by CircleCI.
Our first step was to use Sauce Labs solely to run End-To-End Capybara Tests during our deployment process to ensure we didn’t introduce breaking changes into our closely integrated system.
- End-To-End Test sends Capybara commands (via WebDriver) through Sauce Connect to Selenium Grid
- Selenium Grid interprets commands and drives actions on the Browser
- Sauce Connect proxies the Browser request to EC2
- Instances send data back to Selenium Grid via Sauce Connect
- Selenium Grid interprets data and drives actions on the Browser
- Sauce Connect proxies the Browser response back to the End-To-End Test
How we Debug
The Sauce Labs Dashboard allows us to access information about each test run. We are able to utilize Sauce Labs Screencasts and Selenium Logs to troubleshoot our tests. Screencasts show a video representation of our tests in action running on the different browser + OS configurations. Selenium Logs allow us to easily view the Selenium commands being ran from each test.
But Wait, There’s More
As we got Sauce Labs more deeply incorporated into our product teams and their development process, teams needed to run new types of tests: Protractor and Nightwatch tests. These tests are a lower level than our end-to-end tests and are run against a local mock server to increase speed and reduce flakey tests for more predictability. With Sauce Connect Proxy, a proxy server that opens a secure connection between the Sauce Labs VMs running our tests and the application being tested on the local machine, we are able to proxy requests from Sauce Labs to the local mock servers. We are running these functional tests across 10 browser and OS combinations.
Measuring Success
To track the success of our test strategy and approach, we aimed to collect a few KPI’s of our automated suites. It’s important to highlight these for each one of our product teams to give them a quick glance at how their test suites are progressing (positively or negatively) over time. We measure and graph the following:
- [Regression of Test Suite] Time for Successful Test Runs
- [Health of Test Suite & Code Quality being Committed] Daily passing % for Test Runs
- [How Flakey Tests Are] Number of Test Retries Per Day
The ultimate goal in having a fast, scalable test infrastructure is to increase our developers’ confidence that they are shipping high quality code. Our system does exactly this — the speed and configurability of our tests lets us maintain a tight feedback loop of quality, resulting in happy developers and amazing code.
And finally, by gathering and surfacing the metrics from our test infrastructure, we can keep our product teams posted on the status of their test suites and ensure we are holding our test suites and test infrastructure to a higher standard.
