Using Puppeteer to Create an End-to-End Test Framework
The Uptake Accounts application has grown, in the past year, from being a central location for users to log on to the Uptake Platform to a fully featured tool for managing users, roles, and tenants. As one of the first web applications to join the Uptake Platform, Accounts often gets to be an early adopter of new design components, technologies and practices.
Accounts has always had a solid JUnit test suite for its Java back end and Jest test suite for its Vue.js front end. It’s easy to write tests with these frameworks because they are lightweight, reliable and fully featured. There’s a pretty solid formula to writing them, and most developers can tell a good unit test from a bad one.
We then began thinking about adding end-to-end tests. While unit tests are great for testing a small unit of work, making sure that it is correct in every edge case that could potentially occur, end-to-end tests can assert that a user on the web application can do a sequence of actions with the results we expect. They actually render and interact with the Accounts web application and confirm that every layer of the application is working to deliver the experience we want for the end user.
As a result, these tests have some well-known problems. As Martin Fowler’s test pyramid article calls out, “tests that run end-to-end through the UI are: brittle, expensive to write, and time consuming to run.” So although they had the potential to give our team more confidence that our code was correct, they also had the potential to make our test suite much longer and less deterministic.
This made it very important to be thorough when picking our end-to-end testing solution, since we found that many of the assumptions we could make about the ease of working with unit test frameworks did not apply to many end-to-end test frameworks. What we were looking for was a reliable and fully featured solution that wouldn’t require much effort to integrate into our already solid test suite.
Enter Puppeteer
The Headless Chrome API has existed for a while now as a way to interact with web pages with the Google Chrome engine without using a GUI, but a command line tool doesn’t integrate well with a Javascript code base and isn’t reliable enough for a test framework.
This is probably why the Chrome team bundled their Headless Chrome API into a Node package with a fully fledged API that’s callable from Javascript called Puppeteer. This includes navigating and rendering pages, interacting with page elements, taking screenshots, and executing scripts and adding style sheets within the page’s context.
It was very easy to integrate into our project, all we had to do was add the dependency to the node package. Although it’s not a perfect solution, as some corner cases do require complicated Puppeteer code to properly test, it checked our boxes enough to start writing end-to-end tests in it.
Puppeteer Basics
Puppeteer is basically a programmable browser. So in order to write a test with it, you need to know how to tell the browser what to click, type, or scrape so you can assert some behavior is happening or not happening in your web application.
CSS Selectors
The main way to tell Puppeteer what element to click, type into, scrape, etc. in the DOM (Document Object Model) is to use CSS selectors. A good resource for explaining the syntax can be found here, but if you’ve worked with CSS in any capacity, chances are you know some of it already.
It’s also very easy to go on a page you’re about to test with the Chrome browser, and test out the selectors you’ll need to include in your Puppeteer test. For example, use Google Chrome to open this great demo site for web scraping that looks like a time machine to 1998.
Imagine you want to write a test that makes sure the first link in the top section has “SIGN-ON” as its text and has the appropriate link. You can use Chrome’s developer tools to find this element’s CSS selector that you can then use in a Puppeteer test.
Let’s start by using your keyboard to enter “Inspect Element” mode (Ctrl + Shift + C on Windows/Linux, Cmd + Shift + C on Mac) and click on the “SIGN-ON” link in the top section.
The entire CSS selector path can be seen under the source in the “Elements” tab, but we only need enough of it to specify exactly the element we want. We can see that the parent of all the links in the top section we’re interested in is a tr element, followed by a td.mouseOut element (tr as the element type, mouseOut as the class), then finally the a element that actually contains the link.
Go ahead and open the console, either by finding the “Console” tab next to the “Elements” tab you’re currently viewing or using the keyboard shortcuts from the page (Windows and Linux: Ctrl + Shift + J. On Mac: Cmd + Option + J) and type $$(‘tr > td.mouseOut > a’) and hit enter. What you’ll get is the 4 links from the top section. We want to test out specifically the first link and make sure it says “SIGN-ON”, so a nice pseudo-class selector called first-child will help us get just the first td child of the tr element.
Type out $('tr > td.mouseOut:first-child > a’).innerText and hit enter. Lo and behold, the console tells us that “SIGN-ON” is the text of that element. Browsing the types of CSS selectors and testing them out on the Chrome console can really make it easy to find and interact with elements on the pages you want to test.
Now that we found the selector we need, we can assert the same behavior in a Puppeteer/Jest test. If you already have Jest and Puppeteer configured as dependencies in your project, the following code will run and pass.
import puppeteer from "puppeteer";
test('sign-on link', async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('http://newtours.demoaut.com');
await page.waitForSelector('tr > td.mouseOut:first-child > a');
const firstLinkText = await page.$eval('tr > td.mouseOut:first-child > a', node => node.innerText);
expect(firstLinkText).toEqual('SIGN-ON');
});Methods of Interest
The Puppeteer API allows one to describe a series of steps to execute in a browser context. This is a huge API to explore, including options for saving pages as full PDFs, emulating individual key clicks and adding a stylesheet to the page.
Below, I included explanations of the operations that are most useful for an end-to-end test. You can see each of them in action with this official example from the Puppeteer team that does a search on the Google Developers Portal and prints the links and titles of the results to the console.
goto
This is how one navigates to a specific url in Puppeteer. It is analogous to typing a url into the address bar of a browser.
await page.goto('https://developers.google.com/web/');waitForSelector
This method allows one to wait for a certain element to render after doing an action such as navigating to a page or submitting a form.
In the example below, after typing into the search box, the browser is told to wait until an element appears with the devsite-suggest-all-results class.
const allResultsSelector = '.devsite-suggest-all-results';
await page.waitForSelector(allResultsSelector);type
This method also uses a CSS selector to an element that one can type into, such as an <input type=“text” …> element, as the first argument, and the string to type into the element as the second element. In the example below it’s used to execute the search.
await page.type('#searchbox input', 'Headless Chrome');click
This method is used to click on an element specified by a CSS selector. In the example below it clicks the “Show all results” button on the Google developer page.
const allResultsSelector = '.devsite-suggest-all-results';
await page.waitForSelector(allResultsSelector);
await page.click(allResultsSelector);$eval
This method doesn’t show up as page.$eval in the search example but that’s because there are many ways to evaluate a script within the page’s context (in the search example, page.evaluate is used instead of page.$eval).
One of the most important ways we use $eval in the Accounts end-to-end tests is for getting the text content of an element. For example, you can imagine a test that uses this method to get the text content of the h1 header of the Google developers page in the following way:
const titleText = await page.$eval('h1', node => node.innerText);
expect(titleText).toEqual('Build anything with Google');Walkthrough of an Example Jest/Puppeteer Test File
I think the best way to show how Puppeteer can help test the end-to-end functionality of a web application is to show how a real team on the Uptake Platform team is using it. So I’ll walkthrough an actual end-to-end test that the Accounts team uses as part of our test suite, as well as some of the supporting code that goes with it.
Puppeteer is often used for testing, but that’s not its only purpose, so it does require a little setup to get to the point where it’s ready to be used inside of a test suite. First I’ll briefly go through some of the supporting functions that are required for the test and then I’ll explain one of the tests in detail.
The ActionStack class
One thing that was discovered about using Puppeteer in Jest tests early on was that when some Puppeteer methods fail or time out, Jest will often just throw a generic message about timing out but not tell you what line it happened at. Another thing that was discovered early on was that it was pretty hard to tell what exactly was being tested in a test containing a bunch of page.click and page.type commands on elements with esoteric selector statements.
This is why we introduced the ActionStack class which allows us to describe each action that happens so that:
1) it is clearer to the reader of the code what is happening and
2) if the test fails, the error message says what the action was that failed instead of a generic timeout/Puppeteer API error, making the test much more debuggable.
export default class ActionStack {constructor() {
this.actionsSoFar = [];
}async executeAction(actionDescription, action) {
try {
this.actionsSoFar.push(actionDescription);
return await action();
} catch (e) {
this.actionsSoFar.pop();
let errorMessage = `Failed during action "${actionDescription}", due to error: ${e}\n`;
errorMessage += 'Actions leading up to failure:\n';
for (let i = this.actionsSoFar.length >= 3 ? this.actionsSoFar.length - 3 : 0;
i < this.actionsSoFar.length; i++) {
errorMessage += '"' + this.actionsSoFar[i] + '"\n';
}
throw new Error(errorMessage);
}
}}
On one line for example we have the statement,
page.click('.up-ds-modal__footer-buttons > .up-ds-button--primary-base')which on it’s own makes no sense to someone reading the test for the first time, but
await as.executeAction('Click the submit button', () => page.click('.up-ds-modal__footer-buttons > .up-ds-button--primary-base'));gives more context. And if the click fails, due to the button still being disabled because a required field is missing for example, the error message will tell you what action failed as well as the previous three actions that led up to it, making debugging much simpler whether locally or on a CI/CD server.
Setup And Teardown
Let’s examine how we set up and tear down in an end-to-end test. We can start by looking at the beforeAll block that launches Puppeteer.
beforeAll(async () => {
jest.setTimeout(10000);
browser = await as.executeAction('Launch Puppeteer', () => LiveTestUtils.launchPuppeteer());
});We set the timeout for Jest to be 10 seconds because it’s just long enough to be resilient without taking too long to fail, making our test suite needlessly long. If you want to see what the Puppeteer-controlled browser is doing to help debug an issue, or just because it looks cool, you can add an argument so that the browser is actually visible (keep in mind this only works locally):
browser = await as.executeAction('Launch Puppeteer', () => LiveTestUtils.launchPuppeteer({args: headless: false}));You’ll want to launch Puppeteer in the beforeAll instead of the beforeEach to make the suite go faster if you have multiple tests, since launching and closing Puppeteer is pretty slow. We therefore also close Puppeteer in the afterAll block.
afterAll(async () => {
await as.executeAction('Close Puppeteer', () => browser.close());
});We then start the beforeEach block, where we set up our browser context as an incognito context, which basically means the tests don’t share any cookies or cache with the other tests. The next step is a function that will create our test user that we will use to sign in with using regular Axios requests to the Accounts backend.
beforeEach(async function () {
incognito = await as.executeAction('Open new incognito browser context', () => browser.createIncognitoBrowserContext());
page = await as.executeAction('Open new browser', () => incognito.newPage());
const testUserObject = await as.executeAction('Set up test user', () => LiveTestUtils.setupTestUser());
testUser = testUserObject.testUser;
testUserSecret = testUserObject.testUserSecret;
});This is an important principle in these end-to-end tests, which is to test as little as possible using Puppeteer. If you can send a rest request to do the setup for the test, it’s much more reliable and faster than to try to setup all the mouse clicks and forms to create a test user using Puppeteer.
In the afterEach block, you’ll want to close the context and page, as well as do any cleanup of the entities you’ve created.
afterEach(async () => {
await as.executeAction('Clean up the test users', () => LiveTestUtils.cleanupUsers());
await as.executeAction('Close the page', () => page.close());
await as.executeAction('Close the incognito browser context', () => incognito.close());
});Walkthrough of a Real End-to-End Test
We’ll look at the “Change password” end-to-end test. What this test will do is log in as our test user, change their password, logout, and then login with the new password.
First, we do some REST requests to make it so that the test user can RESTfully agree to the terms and conditions instead of doing it through Puppeteer.
test('Change password', async () => {
await LiveTestUtils.setupTestUserAgreements();We go to the Accounts login page and use waitForSelector to wait until the first element we want to interact with has loaded. Normally this is sufficient for ensuring the page has loaded and is the practice throughout the test suite after navigating to a page or triggering a change in the DOM.
await as.executeAction('Go to the login page', () => page.goto(LiveTestUtils.accountsUri + '/login'));
await as.executeAction('Wait for the login page to load', () => page.waitForSelector('input[placeholder="Username"]'));We then type our test user’s code and password and click submit.
await as.executeAction('Type the user\'s username in the username field', () => page.type('input[placeholder="Username"]', testUser.code));
await as.executeAction('Type the user\'s password in the password field', () => page.type('input[placeholder="Password"]', testUserSecret));
await as.executeAction('Click the submit button', () => page.click('#login-submit'));We wait for the first element we want to interact with again, this time the title of the user’s profile and scrape it’s text to make sure it includes the test user’s code.
await as.executeAction('Wait for user\'s profile to load', () => page.waitForSelector('.breadcrumbs > .up-ds-type--thin-heading-xxs'));
const textFromProfileTitle = await as.executeAction('Get the text from the title of the user\'s profile', () => page.$eval('.breadcrumbs > .up-ds-type--thin-heading-xxs', node => node.innerText));
expect((textFromProfileTitle).includes(testUser.code)).toBeTruthy();We then test out the cookies to make sure they include an Authorization cookie, which means that the Puppeteer context is actually logged in as the test user.
const profileCookies = await as.executeAction('Get cookies', () => page.cookies());
expect(profileCookies.filter(cookie => cookie.name === 'Authorization').length).toEqual(1);We then click the combo button and navigate to the “Change Password” option.
await as.executeAction('Click the combo button', () => page.click('.up-ds-button--menu'));
await as.executeAction('Click the "Change Password" option', () => page.click('.up-ds-dropdown-option__anchor'));First we select what our new password will be. We’ll use a type 4 UUID to make sure it’s a completely random string. We’ll then type out our test user’s old secret and then enter and confirm their new password and hit submit.
const updatedSecret = uuid();
await as.executeAction('Type the user\'s old password', () => page.type('input[type="password"][placeholder="Enter current password"]', testUserSecret));
await as.executeAction('Type the user\'s new password', () => page.type('input[type="password"][placeholder="Enter new password"]', updatedSecret));
await as.executeAction('Confirm the user\'s new password', () => page.type('input[type="password"][placeholder="Confirm new password"]', updatedSecret));
await as.executeAction('Click the submit button', () => page.click('.up-ds-modal__footer-buttons > .up-ds-button--primary-base'));From there we’ll navigate the browser to the logout page and confirm that the browser’s authorization cookie is gone, meaning the browser is actually logged out.
await as.executeAction('Go to the logout page (which will navigate to the login page)', () => page.goto(LiveTestUtils.accountsUri + '/logout'));
const loginPageCookies = await as.executeAction('Get cookies', () => page.cookies());
expect((loginPageCookies).filter(cookie => cookie.name === 'Authorization').length).toEqual(0);We’ll then follow the same process as before to login.
await as.executeAction('Wait for the login page to load', () => page.waitForSelector('input[placeholder="Username"]'));
await as.executeAction('Type the user\'s code in the username field', () => page.type('input[placeholder="Username"]', testUser.code));
await as.executeAction('Type the user\'s updated secret in the password field', () => page.type('input[placeholder="Password"]', updatedSecret));
await as.executeAction('Click the submit button', () => page.click('#login-submit'));And finally we’ll test that we’re logged in by confirming that the page we’re on has our test user’s code in the title.
await as.executeAction('Wait for the user\'s profile to load', () => page.waitForSelector('.breadcrumbs > .up-ds-type--thin-heading-xxs'));
const textFromProfileTitle2 = await as.executeAction('Get the text from title of user\'s profile', () => page.$eval('.breadcrumbs > .up-ds-type--thin-heading-xxs', node => node.innerText));
expect((textFromProfileTitle2).includes(testUser.code)).toBeTruthy();
});If anyone in the Accounts team submits a feature that changes how a user updates their password, this end-to-end test gives us confidence that we can confirm we haven’t broken anything or that we’ll catch the error before letting the feature be merged.
Puppeteer Use in the Future
Our end-to-end test suite buys us a lot of security. When any of us submit a pull request, we can confirm that our changes haven’t broken the Accounts application being able to launch against our development environment, or from being browsable by a Chrome browser, on top of the individual guarantees each test provides (that you can login, change your password, etc.).
We probably won’t add too many more end-to-end tests though, since they’re slow and less reliable than unit tests. However, there are plenty of other uses for Puppeteer. One idea is to try to leverage some of Puppeteer’s other features to create documentation that’s never out of date by adding end-to-end tests that take screenshots and write documentation as they run.
Hopefully more teams will leverage Puppeteer for their end-to-end tests as the Platform develops and all of our web applications will benefit from the assurances they bring.
