If you can measure it you can improve it

In today’s R&D ecosystem, in which new features and lines of code are released daily (and sometimes many times per day), our code-bases, and their attendant complexities, grow continuously.

We prevent breakage during this precipitous growth by writing unit tests, designing smart automation flows, executing manual QA tests, and more.

But what about performance? do we know if our new features precipitate performance regressions or memory leaks? Can we pinpoint exactly where and when CPU usage gets high? With Puppeteer — YES!

In this post you will learn how to build your own automated performance record and alert system, just like the one we have at WalkMe.

How does it look?

Using an automated performance record and alert system, you can obtain stats that tell you which component(s) in your application improve or regress with respect to various performance metrics. These stats are presented like so:

You can even track performance over time:

A dashboard with Heap and DOM elements-count visualizations for the OpenWidget flows

Ready to showcase your performance stats? Read on!

The tools

Enter Puppeteer

Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

Puppeteer enables you to automate almost every manual action in the browser, such as page navigation, buttons clicks, screenshots, and more. Puppeteer with Jest integration further enables you to write UI and end-to-end tests for short user scenarios such as “add item” or “open menu.”

Configuring Puppeteer and Jest

To integrate Puppeteer and Jest, you must add the following NPM package:

npm install --save-dev jest-puppeteer puppeteer jest

And update your Jest configuration:

{
  "preset": "jest-puppeteer"
}

It is simple to write end-to-end flows with Puppeteer, and it is similar to translating manual actions to API calls:

You can view a step-by-step tutorial on the official website.

Measuring performance metrics with Puppeteer

What are we measuring ?

By using the Puppeteer API (page.metrics()), we can inspect and track performance metrics such as DOM node count, memory consumption, CPU load, etc.

How are we measuring?

The procedure is as follows:

• Measure metrics from the page.
• Perform the tested action.
• Measure metrics from the page again.
• Compare the after metrics to the before metrics.

To increase accuracy, perform garbage collection before each measure.

Let’s implement the four steps:

(1): Collect garbage (to get more accurate data) and measure metrics from the page.

(2): Execute the given delegate function. this test function should evaluate the actions we want to measure.

(3): Collect garbage and measure again, but this time, only measure the metrics after the evaluated action(s) has/have been performed. Now we have two metrics: before and after the evaluated action(s).

(4): Calculate the differences between those two metrics.

The measure test function — get metrics before and after evaluating the test function; calculate diffs

See all the measures we take in part (4) of the script above? We will store these and analyze them later on.

Now that we have our measure function, let’s use it with Jest:

Use the measure wrapper inside the Jest test

In the above test, we called the measure function and sent it the action to perform.

In part (2), you can add more data with such metrics as branch revision, environment, and/or other data that makes analyzing problems easier.

Tracking over time — Where can we save the data and how can we see timeline changes?

We can log the metrics to a local database such as SQLite, and create tests with thresholds.

First, we need to create the measures table. this table will contain an ID, the test name, and the measures raw data:

Create a new measure table SQL statement

Then we will create the dbHelper file, which will handle the DB connection and expose two functions: reportData, which receives new measures and inserts them into the table, and getLastRow, which retrieves the last measures of the given test.

Then, we can add the DB reporting after taking measures:

And finally, we will create a function that will measure the metrics from the page, and report them to the DB:

Now that we know how to record and report performance data, lets move on to building and creating a threshold. We need to add the threshold tests that verify that we don’t pass threshold values.

Summary of steps: First, navigate to the test page. Then, wait until you see the menu button. Next, open the menu. Finally, compare the measures diffs.

Open a menu test

Here is an example of a test:

First, let’s take the last measures of the tested function. Next, we’ll use the measureAndReport function that we wrote to measure metrics of the tested function and report the results to the DB. Finally, we’ll execute tests to verify threshold exceptions:

And here are the Jest output results:

Jest output results

Now we can connect the tests to our CI platform (Team City / Jenkins, etc.), trigger to every commit, and start committing with no worries ;).

Visualizations and alerts

A good start to data visualization is to use DB Browser for SQLite to see the data entries and view some plots:

A DB Browser from DBBrowser for SQLite. We can see the data and basic plots

In addition, we can send the metrics to any database, such as Logz.IO, and then create dashboard graphs with Kibana or a number of other dashboard makers. At WalkMe, we send our metrics to Logz.IO and visualize the data with Kibana.

A dashboard with Heap and DOM elements count visualizes of the OpenWidget flows

We can define specific alerts that will notify us when we have a regression. We can also configure the sending of emails or messages via Slack when there is an exception value.

Specific metrics status dashboard

We can even report notification alerts to Slack:

Informative alert notifications in Slack

Additional awesome stuff that can be done with Puppeteer

As you see, we can measure performance with Puppeteer. But what specifically can we measure?

• We can measure the loading time of components. E.g., we can measure the time lapsed from the moment the end-user clicks a button until the component is shown.
• We can verify linear stable memory and CPU usage of components.
• We can recognize memory leaks or unreleased event listeners. E.g., investigate a clear button that removes items from the page.
• We can recognize UI problems, e.g., redundant reflows.

In addition to measuring performance, we can also perform end-to-end UI tests such as the following:

Image Compare

Puppeteer allows us to easily take screenshots of a page, so why not use this capability? we can very simply run an image comparison test to verify that we haven’t change element style or broken the UI.

We will use pngjs to handle reading images, and the pixelmatch package for comparing image differences.

Example of an image compare test:

DOM tree compare

With Puppeteer and Jest we can compare DOM structure in order to verify we haven’t broken the UI, changed classes, IDs, node hierarchy, etc.).

To this end, we can use thetoMatchSnapshot function from Jest. This function saves the DOM content in a folder, and, in the subsequent run, it compares the new DOM content with the previously saved content.

Real life example — struggling with JScrollPane

Scenario: We received an email alert stating that we had passed the allowable threshold of certain metrics. We started to investigate it, and, by looking at the graphs, we saw that there had been a dramatic growth of memory, duration, recalculate style, and DOM count.

Troubleshooting: We focused on the graph values at the first point where the values started to grow dramatically, and, using the branch name and commit hash (which exists on the measure data), we bisected and found a rebellious commit.

It turned out that one of the developers used a third-party library called JScrollPane (an old scrollbar library), and this was causing the dramatic growth of the numbers.

Note: Every time your code changes, you will see a performance impact. It might be a positive change, or it might be a regression. Before taking any action, decide whether the performance impact is acceptable or not, and whether you must fix it.

Solution: Of course, we couldn’t accepted the growth that occurred in our example, so we changed the integration with the library and denounced redundant calls, and, by doing so, lowered the measures 👏.

Before you start writing end-to-end performance tests, consider these points:

• These performance measures are the result of lab tests, not real-life experiments. In real life, end-users work with multiple tabs on many instances of web browsers, all while using various hardware. As such, we can’t predict exact behaviors or results in edge-cases.
• Unfortunately, Puppeteer is only supported on Chrome, so other browsers can’t be tested in this way.
• I recommend using alerts that notify you via email or Slack (or other correspondence platform) when there are any exceptions.

Summary

In this article, we saw how to use Puppeteer in order to measure performance metrics. We also saw how we can store them in a DB and visualize them. Finally, we learned how to create alerts through tests we all know and love to write (right?…).

But, believe it or not, this is only the tip of iceberg! There are many cool things you can do with Puppeteer. If you encounter additional awesome Puppeteer scenarios, let me know and I’ll add them if you say “pretty please.”

At WalkMe, we use various tools in our CI/CD procedure in order to ensure a high quality of product delivery. Nowadays, automating such processes is a must in order to keep up our pace of development while ensuring optimal UX is maintained. Puppeteer is but one tool in our arsenal, and we are constantly developing more tools and methodologies in order to deliver better and faster.

Many thanks to Moran Ezra for his technical guidance and help writing this post, and thanks to Aaron Weiss for proofreading it for me!