How we built better automated UI testing at Vimeo

Automated UI testing is an important part of evaluating the quality of a given software build or release candidate. By running tests against a real version of our application at Vimeo, we can very closely mimic the end-user experience. This is done to increase quality and confidence in our builds by ensuring that key components are working as intended, which helps our human QA testers to focus on things that humans are good at: contextual and exploratory testing. We have some of the best human QA testers in the business, but nobody wants to spend hours making sure every button on the Vimeo Player works with each new release. Reducing the burden of regression testing frees up our QA analysts for deeper, more complex UI tests.

At Vimeo, we wanted to enable automated UI testing and roll it out across teams. The idea here is that different teams need automated UI testing in slightly different ways, yet everyone ought to be using the same overall framework. So we wanted to split the work between types of engineers, where application engineers write the tests and scenarios, and automation engineers maintain the core framework. This way, automation engineers can support and guide test writing but not maintain the suites and scenarios themselves, while application engineers, who understand the product, can write great tests but not worry about implementing and maintaining automation frameworks.

We accomplished this by building a Node.js package called the UI-Tests-Backbone. Backbone settles dependencies (chiefly WebdriverIO and its sub-packages) and provides utility functions and browser configurations that are useful to teams at Vimeo in a generic way. While different teams have different application functionality to test, these generic Backbone tools (such as headless authentication and shared configuration data) are useful to all teams; that is, they aren’t team-specific. In addition, we created the UI-Tests-Playbook, our detailed documentation, along with a boilerplate code repository to get started. By providing this for all teams, any application engineer can begin creating a UI testing suite for their team without having to research, evaluate, or experiment with setting up a framework on their own.

The following screenshot shows one of Backbone’s more important functions, baseAfterTest, which records test results in the report after a test finishes.

It’s implemented in the WebdriverIO afterTest hook, like so:

Now, when a test fails, the generated report is standardized to include helpful information to diagnose the problem. The figure below shows a failing test. The report includes a link to the Browserstack session (which features a video recording), as well as a screenshot, browser logs, and the entire HTML document at the time of failure for easy reference.

One challenge we ran into along the way was finding an optimal method of packaging and distributing Backbone so it could be installed as needed in our Jenkins environment. npm (Node Package Manager) enables you to specify GitHub URLs, which is a convenient way to deliver Backbone as a dependency to various projects. However, at Vimeo, we use a private GitHub server for all of our code, so when it came time to run tests in our Jenkins environment, it wasn’t clear how we could accomplish package installation for our private Backbone repository. Fortunately, at Vimeo we have an amazing DevEx team (as in Developer Experience) that helps keep us moving by providing robust tools and processes for all of our engineering teams. They had already made the Credentials Binding Plugin available for our internal developers, and by wrapping our npm install step with the plugin’s withCredentials block, installation was a breeze.

To ensure the success and adoption of the Backbone project, we provided additional tools and video supplements. We used Vimeo Record to make instructional videos and Vimeo Create to promote the tool internally. Finally, we used Hubot to integrate test runs with Slack (you may have heard of it, we use it at Vimeo for in-house chat and communication), so engineers and non-engineers alike can run tests and get results without needing a development environment. Just a simple command in Slack kicks off tests and delivers the results to the user, lowering the barrier for entry.

Backbone was a challenging yet effective project for facilitating and expanding the scope of QA Automation at Vimeo. So far, five teams have implemented new suites of tests, and we’ve seen several dozen manual test runs via Slack. This is great, but developers also want these tests to run automatically as code changes are made. And with that, we are currently developing the means to run these tests automatically as part of our Continuous Integration software development cycle.

Interested in flexing your engineering chops at Vimeo? Join our team!