Strava’s New End-to-End Testing Setup

Strava runs end-to-end tests of our website before every deploy, but problems have developed over time, mostly around reliability and the ease of adding more tests. Until recently, we ran a suite of Selenium tests against an older version of Firefox (which was compatible with some custom testing plugins). During our last Web Guild Week we decided to rewrite our test suite using new tools and aimed for reliability, simplicity, and utility. We made great strides and now we run an entirely new test suite.

At the outset, our main motivator was reliability. Sometimes our end-to-end tests failed for mysterious reasons even though the app was fully functional. Or, they would pass despite a crucial, undetected error. Developers came to regard the QA tests as an impediment rather than a tool. Every inconsequential red failure demanded attention while the green successes remained untrustworthy.

Beyond that, we wanted to upgrade the technology behind our tests. Our old QA suite only ran in an older version of Firefox. We wanted to run our tests in more browsers, or at least in a more modern browser.

Historically, developers skipped writing end-to-end tests mainly because those tests were hard to write and seemed only to make the system more brittle. Web developers at Strava write many Ruby unit level tests and we have started writing front end tests using Jest. We wanted to make end-to-end tests as easy to write as our unit level tests.

Our new testing tool

Cypress is a newer end-to-end testing tool. It competes most directly with Selenium. Tests in Cypress can be written in JavaScript. For a web developer, this makes more sense than writing tests in Ruby because web browsers themselves run JavaScript and our web app runs in browsers. Navigating the DOM using JavaScript, sometimes with jQuery, is more familiar than with Ruby.

At its most basic, a test looks like this:

beforeEach(() => {
  cy.fixture(‘athletes.json’).as(‘athletes’);
});
it(‘logs in’, function() {
  const athlete = this.athletes.free;
  cy.visit(‘/login’)
    .get(‘input[name=email]’)
    .type(athlete.email)
    .get(‘input[name=password]’)
    .type(athlete.password)
    .get(‘#login-button’)
    .click()
    .url()
    .should(‘match’, /onboarding|dashboard/)
    .get(‘.athlete-name’)
    .should(‘contain’, athlete.first_name);
});

Our Selenium tests ran with an older version of Firefox that we patched to notify us of JavaScript errors. This became difficult to run locally. Cypress automatically boots its own version of Chrome, with some custom tooling on top of the familiar Chrome experience, and it automatically fails when it sees a JavaScript error.

We are using Cypress’ open source tool, installed as an npm package. Cypress offers a paid ‘dashboard’ tool that we have not yet investigated.

Our goal during Guild Week was to port our old Selenium tests to Cypress. These old tests covered a wide swath of functionality and we translated them line by line.

The old test might look like this:

client(:free).find_element(css: ‘.page .profile’).click

In this Selenium test, client(:free) has logged in and remains logged in as an athlete on the ‘free’ plan. In Cypress, we don’t use “clients” this way, instead we actually log in as necessary, usually with a freshly created new user. Using Cypress, the code looks like:

cy.login(athlete.email).get(‘.page .profile’).click()

Cypress will remain logged in until we log out or log in as a different user.

Continuous Integration

Strava uses an internal continuous integration service named Butler for our deployment processes. You can read more about Butler here.

We deploy our Web app by instructing Butler to cut a new deploy branch and run our entire test suite against that branch. We decided to include the new Cypress tests in this deployment cycle, alongside our existing Selenium tests.

We took advantage of Cypress’s built in screenshot and screencast capabilities. Whenever a test failed we would send the image and video artifacts to an AWS S3 bucket. We then generated links to those artifacts that we could display in our CI web UI. Lastly, we instructed Butler to send green / red, success / failure updates to our Slack channel.

We also took advantage of Cypress’ extensibility, because adding new Cypress commands is easy. For example, we added a `cy.login` command:

Cypress.Commands.add(‘login’, () => { … })

We added several other utility commands with the simplest names we could think of:

cy.login(email, password)
cy.register().then((newAthlete) => { … }
cy.fillOutCreditCardForm(…)

Generally, a function that we use only in a single file stays in that file. Otherwise, when we want to re-use a function across 2 or more files, we create a new command.

Eslint

We use Eslint to automatically detect JavaScript style issues in our web application. Cypress itself uses eslint, and can take a separate cypress-specific .eslint configuration file that works only in the Cypress folder. We have made some custom eslint configurations that differ from our production app.

• Functions: arrow vs regular

We generally avoid arrow functions in our Cypress tests because they lose context when using fixtures. Instead, we use normal functions that are bound correctly to the `this` context. We added a rule to our eslint to allow this:

“prefer-arrow-callback”: 0

• Anonymous functions

Using traditional function declarations, we do not want to provide a name for those functions. Naming the function does not provide enough benefit to outweigh the messiness with the test. We want to allowed unnamed anonymous functions, so in our eslint, we have:

”func-names”: 0

• Extends our eslint-config

Beyond those 2 rules, we want to use the same settings as the rest of our app, based largely off airbnb’s eslint config:

“extends”: “@strava/eslint-config”

Lastly, Cypress provides its own eslint plugin, so that we can use `cy` and other globals without incurring eslint errors. We added to our eslint config:

“plugins”: [“cypress”]

Server Side Code

Historically, we accomplished many mundane test steps via the UI. We would have a user login via the login form on the login page, or we would have the admin verify users via our admin backend. But now, we have special routes that bypass the UI entirely and execute code on the backend. It’s much faster to create a new user via backend code than it is to drive the web browser through the user registration flow. Also, this allows us to use fresh new users for every test, instead of relying on persistent user fixtures.

Usually, when an athlete registers a new account via the web browser, we ask that they verify their email address. But in a test, we want to verify the new test athlete’s email right away, without blocking the test.

We added a cy.verifyAthlete command to quickly verify a newly registered athlete via backend code:

Cypress.Commands.add(‘verifyAthlete’, (email) => {
  cy.request(‘POST’,’/cypress/verify’, { email }
});
/cypress/support/commands.js

Then, we added a route to handle this request. Note, we make sure this route is only available in a VPN protected QA environment:

post ‘/cypress/verify’ => ‘cypress#verify’
/config/routes.rb

And a controller action:

# /app/controllers/cypress_controller.rb
class CypressController < ApplicationController
  def verify
    # Find and update the athlete
  end
end

This pattern — add a Cypress command, a server side route and the controller action — allows us to avoid using persistent fixture data. Instead, we can setup data at any point through backend commands.

Conclusion

At Strava, we now run all our end-to-end tests using Cypress. We ported our old suite of Selenium tests to Cypress and write new tests for new features. Nowadays, when we see a red failure, it usually means we have to fix something. When it’s green, we deploy our app confidently and quickly.