Improving End-to-End Testing at Coursera using Puppeteer and Jest

Billy Kirk
Jun 26, 2019 · 6 min read

By Ankit Ahuja, Billy Kirk, and Sumit Gogia.

Coursera is committed to delivering a high level of product quality for every learning experience. This includes individual courses, and courses included within Degree & MasterTrack™ programs.

In the past, we’ve used Selenium and Sauce Labs for running end-to-end tests. Stability issues with our setup and limited continuous integration (CI) tools around test result reporting led to poor adoption.

To make end-to-end (e2e) testing an essential part of our culture and help deliver a quality experience on the platform, we focused on a few key principles:

Developer Experience

  • Make end-to-end tests simple to run and debug
  • Push test failure notifications to engineers in code review
  • Leverage open source libraries


  • Push test reports to engineers in code review
  • Clearly highlight failures and suite stability through notifications and reporting
  • Assign clear owners to groups of tests


  • Run many tests quickly and reliably
  • Make it straightforward for new engineers to learn

Recently, we revamped our end-to-end testing system using Puppeteer and Jest. We added tools to improve the development workflow, and made sure test failures were actionable. The new system has improved the consistency of our tests, and helped us deploy with confidence faster.

Writing Tests with Puppeteer and Jest

To facilitate a smooth test writing experience for engineers, we wanted our tests to rely almost completely on Puppeteer and Jest APIs. Puppeteer is a Headless Chrome Node API, which controls a version of the Chrome browser. Some highlights of Puppeteer include:

  • Built on top of the Chrome DevTools Protocol
  • Event-driven architecture
  • High quality documentation & strong community of maintenance
  • Ability to test against different page viewports and network conditions

A significant tradeoff we considered during this process was cross-browser testing. We found that many of our bugs were non-browser specific. Given the other methods we currently use for catching browser-specific bugs (not writing browser-specific code, better linting, JavaScript and CSS polyfills, smoke-testing on other browsers we support), we felt comfortable with this decision. Additional considerations included estimating migration time and setting up training for engineers to adapt from Selenium.

We decided to use Jest for describing, running, and reporting, which was convenient for the following reasons:

  • Our unit tests were already written in Jest
  • Great community and high quality documentation
  • Configuration options for running tests in different environments

Additionally, we exposed a set of test utilities that abstract away complex work from our actual tests. These include authentication, emulating different devices and network conditions, interfacing with our A/B testing system, accessibility checks, and accessing user and course fixtures.

Our resulting test files are compact, and read similarly to our unit tests:

Coupled with Puppeteer’s debugging tips, this familiar syntax decreased the time for engineers to write a test. This meant that they spent more time thinking through edge cases in product flows.

Running Tests

We chose cloud tools that would allow us to run many tests in parallel, and integrated well with our existing build and deployment systems. We followed a few helpful examples and commentary when setting up our Puppeteer and Jest integration using AWS Lambda.

Our cloud test runner addressed two primary use cases:

  1. Run all tests against the deployed versions of front-end applications in production.
  2. Run app-specific tests for each pull request and reject if tests fail.

We built a local command-line interface (CLI) using Commander to simplify local test development and debugging. The CLI encapsulates configuration options for both Jest and Puppeteer through one interface. Some example commands include:

In contrast, our homegrown setup required engineers to run a standalone server alongside the local development environment to run tests. This led to inconsistent results when running a test in the cloud. Failure debugging was often inconclusive and non-deterministic, and we frequently mitigated issues by adding retries and increasing timeouts.

Timely and Relevant Notifications

For each pull request submitted for review, our CI process generates a staging environment based on the changes. The end-to-end test suite runs against the apps that changed. A notification posts to our code review tool (Phabricator) when the test runs are complete.

Additionally, we set up test group configurations, which define tests corresponding to a given suite. Based on the group configurations, the framework can ignore certain tests, report separate results for different teams, and notify group-specific channels. Furthermore, test results are still visible in an aggregated summary view. These features have enabled us to add new tests without causing noise within our production suite.

Facilitating Debugging With Clear Reports and Tracing

We also wanted to clearly report results to make it easy for engineers to identify, diagnose, and resolve test failures. All end-to-end test jobs generate a report, which details the results of all the tests run for a given job. The top of the report highlights failed tests, and each test block contains the results of each assertion. We expose a stack trace for failed assertions, along with a command to run the test locally against the given version.

When building features, engineers frequently use Chrome DevTools to capture a timeline trace of a page. We leveraged the timeline viewer to capture a snapshot of each test failure to give engineers a head start when debugging a test failure.

Addressing Flakiness by Monitoring Tests

Test flakiness remained a persistent issue. Common causes of flakiness included test concurrency, setup/cleanup of state, and API timeouts. As a baseline approach to combating flaky tests, the new framework runs each test three times. Passing status is calculated as a summary of all the retries.

We needed a way to accurately summarize test runs to get an idea of end-to-end test health. To that end, we built a data visualization that illustrates test results over a 30-day period. The visualization displays the passing health of each test in total as a ratio of successful runs for that test over total runs. A user can drill down into individual tests to view summarized results on a per-assertion basis.

We notify relevant Slack channels immediately when tests fail, so test and app owners can quickly respond. We also set up a weekly summary to increase visibility and bring gaps to light on a regular basis.


Proper metrics are paramount when evaluating the efficacy of a new system. From experience, we’ve found that good metrics are needed for indicating success and driving execution. In some cases, such as our site-wide availability push, the uptime metric accomplished both. With e2e testing, our primary end goal was to capture more critical bugs before they hit production. With this in mind, we wanted to choose metrics that were easily split and assigned to owners.

We are monitoring useful metrics for e2e testing, which include: application weekly pass rate, count of high priority bugs filed, critical bug capture rate, application e2e test coverage, and number of critical product flows tested. Additional next steps for this framework include setting up access to dynamic test data, running tests against back end changes, and implementing visual diffing.

With the advent of many new features augmenting the Coursera experience, this e2e testing framework became a team priority. Writing a full production suite of tests using this new framework has added an additional layer of stability to our CI process. Furthermore, it has helped codify product flows critical to Coursera’s success.

Coursera Engineering

We're changing the way the world learns!

Thanks to Sumit Gogia and Ankit Ahuja

Billy Kirk

Written by

Product Engineer @Coursera, @UVA ‘13

Coursera Engineering

We're changing the way the world learns! Posts from @Coursera engineers and data scientists.

Billy Kirk

Written by

Product Engineer @Coursera, @UVA ‘13

Coursera Engineering

We're changing the way the world learns! Posts from @Coursera engineers and data scientists.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store