Measure the performance of a web application with Lighthouse CI in a GitLab CI pipeline

Published in

TUI Tech Blog

8 min readJun 15, 2021

Before deploying the code in production, a best practice is running all required tests: unit, integration, and E2E tests. But why don’t you also test the performance and the accessibility of your web application?

In TUI Musement, every team implements unit and integration tests and we have a QA team dedicated to the automation of E2E tests, to run on a pre-production environment. So we have decided to add a quality test to maintain an excellent level of performance, accessibility, SEO and best practices for our products.

This guide will show you how to test the performance, accessibility, SEO, best practices, and PWA rules of your web application. In order to do this, we will add Lighthouse CI to a GitLab CI pipeline. Lighthouse CI uses Lighthouse, the tool integrated into Chrome-based browsers.

We will run the Lighthouse job on Merge Request creation and update and commit to the master branch, but you can change this easily, according to your needs.

✅ You can find the complete repository of this guide here: a sample Nuxt app (but you can use any framework you want) with Lighthouse CI and GitLab CI integrated.

Install Lighthouse CI

The first step is installing Lighthouse CI in your web app. You can install it globally or locally as a dev dependency. I prefer the second way:

npm install --save-dev @lhci/cli@0.7.2
# or
yarn add --dev @lhci/cli@0.7.2

⚠️ I used version 0.7.2 of Lighthouse CI, so I am sure that this version works for all parts described below. But you can try the latest 0.x version, and it should also work.

Configure Lighthouse CI

The second step is configuring Lighthouse CI, to do this you need to create a new file called lighthouserc.js in the root folder of your project.

👉 You can also configure Lighthouse CI using YAML or JSON, as described here.

You can find the final version of lighthouserc.js here

collect

The first section of the configuration file is relative to the collect step: how Lighthouse runs the tests and collects the data. You can find detailed documentation here.

The property numberOfRuns indicates the number of runs that Lighthouse CI makes for each URL. Every run is independent of the others. The Lighthouse team suggests running it multiple times to reduce the variability of the results: a good value can be 5.

The property startServerCommand is the command to run the server before Lighthouse runs. In my repository, I used Nuxt and I needed to run the Nuxt server before, so I have used npm start and it starts nuxt start:

In the **package.json** of my Nuxt app, the “npm start” runs “nuxt start”

The property url is the list of the URLs Lighthouse tests; each URL will be tested numberOfRuns times.

Run “npm start” and, after, run Lighthouse on 2 URLs five times each

If you don’t specify the property onlyCategories, Lighthouse CI will test all categories (Performance, Accessibility, Best Practices, SEO and, Progressive Web App) on your web app, unless you specify exactly what categories you want to test. In this case, we are testing all categories except the PWA audits, because our web app will not be a Progressive Web Application:

You can use the property chromeFlags to pass flags to the Chromium browser Lighthouse uses:

You can set extra headers or cookies using extraHeaders: it can be one of the options to run Lighthouse on authenticated pages:

assert

The section relative to the assert step describes which assertions Lighthouse uses and how it manages any failures. You can find detailed documentation here and here.

The configuration of the assertions is very important because a failure can stop your pipeline and, consequently, you cannot merge your branch or deploy your code to production if one or more assertions are not respected.

If you don’t need a custom configuration, you can use a preset (e.g., lighthouse:recommended ), otherwise, you can add custom assertions starting from a preset:

Start from a preset and add some custom assertions

You can set an assertion on:

single audits (e.g., first-contentful-paint );
entire categories (e.g., categories:performance );

For each assertion, you need to set:

a level between off (the audit is not checked), warn (the audit is checked but the error generates only a warning, and the pipeline is not stopped) and error (the audit is checked, the error returns a non-zero exit code, stopping the pipeline), as described here;
a minScore under which the assertion fails;
an aggregationMethod between median, optimistic, pessimistic, and medianRun, as described here.

In this case:

the performance score must be greater or equal to 90 in the best run
the accessibility score must be equal to 100 in all runs
the best practices score must be equal to 100 in all runs
the SEO score must be equal to 100 in all runs

👉 You can find:
- a description of how assertions work here;
- a list of performance, accessibility, SEO, best practices, PWA audits;
- and a practical list of audits and categories.

upload

The last section upload indicates where Lighthouse saves the report data. You can find detailed documentation here.

You can choose to save them:

on a temporary store with target=temporary-public-storage . You can access the reports by clicking on the links at the end of the test
PROS: easiest and fastest way;
CONS: it’s public and temporary; no history;
on a Lighthouse CI server with target=lhci
PROS: historical archive of all tests;
CONS: time/resources needed to create a server, as described here and here;
on the filesystem with target=filesystem
PROS: save (and download) reports as artifacts in the CI;
CONS: the access to the reports is less immediate than other methods; no history.

We proceed with the easiest way, for now, using a temporary store.

⚠️ The temporary store is public, so take care not to share sensitive data.

Run Lighthouse locally

At this time, you have configured Lighthouse and you can run it locally. You only need to add this new NPM task to your package.json:

Added lighthouse task to **package.json**

Now you need to create your production bundle and run Lighthouse:

npm run build
npm run lighthouse// or with Yarn:
yarn build
yarn lighthouse

It prints something like this:

✅  .lighthouseci/ directory writable
✅  Configuration file found
✅  Chrome installation found
Healthcheck passed!Started a web server with "npm start"...
Running Lighthouse 5 time(s) on http://localhost:3000/
Run #1...done.
Run #2...done.
Run #3...done.
Run #4...done.
Run #5...done.
Running Lighthouse 5 time(s) on http://localhost:3000/detail/1
Run #1...done.
Run #2...done.
Run #3...done.
Run #4...done.
Run #5...done.
Done running Lighthouse!Checking assertions against 2 URL(s), 10 total run(s)All results processed!Uploading median LHR of http://localhost:3000/...success!
Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1621867616820-1661.report.html
Uploading median LHR of http://localhost:3000/detail/1...success!
Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1621867617506-21870.report.html
No GitHub repository slug found, skipping URL map upload.
No GitHub token set, skipping GitHub status check.Done running autorun.

Lighthouse ran npm start using the configuration property startServerCommands ;
when the server was ready, it ran the tests;
in this case, all assertions were valid, unless it returned a non-zero exit code;
it uploads the report on the temporary public store https://storage.googleapis.com .

Run Lighthouse on GitLab CI

It’s time to run Lighthouse on your GitLab CI, creating .gitlab-ci.yml in the root folder.

You can find the final version of .gitlab-ci.yml here

We used cypress/browsers:nodeXX.XX.X-chromeXX-XX as Docker image, as suggested in the official documentation of Lighthouse CI.

We defined only one stage called test and it contains a job called lighthouse:

with allow_failure: false , if it fails then the pipeline fails;
using the property script , it installs all dependencies (npm install), creates the build (npm run build), and runs lighthouse (npm run lighthouse).

We defined a workflow , as suggested here, to define how the GitLab CI pipeline works:

it runs a merge request pipeline when a merge request is created/updated;
it runs a branch pipeline when a commit is pushed on any branch, but there is not a merge request opened on that branch.

In this way, using the rules property, the job lighthouse runs on Merge Request creation/update or commit on the master branch:

Run lighthouse on MR creation/update or commit on the master branch

Create a Merge Request

It’s time to test all creating a Merge Request.
We have a Nuxt application with 2 pages where we fetch a todos API. I created a Merge Request from a branch where I added a fake timeout of 5 seconds before fetch:

Added a fake timeout of 5 seconds to test performance

In this way, the Lighthouse will find a blank page for 5 seconds and we expect the performance score will be decreased.

As we set GitLab CI, the Lighthouse job ran on our Merge Request pipeline… and it fails:

We can discover why it’s failed, by opening the job in detail:

$ npm run lighthouse...Checking assertions against 2 URL(s), 10 total run(s)1 result(s) for http://localhost:3000/ :✘  categories.performance failure for minScore assertionexpected: >=0.9found: 0.83all values: 0.82, 0.83, 0.83, 0.81, 0.831 result(s) for http://localhost:3000/detail/1 :✘  categories.performance failure for minScore assertionexpected: >=0.9found: 0.86all values: 0.85, 0.85, 0.86, 0.85, 0.86Assertion failed. Exiting with status code 1.Uploading median LHR of http://localhost:3000/...success!Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1622222737726-80625.report.htmlUploading median LHR of http://localhost:3000/detail/1...success!Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1622222738383-49315.report.html...ERROR: Job failed: exit code 1

As you can see, there is a failure for 2 assertions: the performance score (0.83 and 0.86) is less than the minimum score (0.9) for both pages. It attaches the link to the Lighthouse report for all URLs analysed:

One of the reports reported above (the reports are saved in a temporary store, so they can expire).

It exited with an error code, and the pipeline cannot continue. We cannot deploy our code to production until we have resolved all failed assertions.

Conclusion

Using Lighthouse CI we can maintain a good quality of our product, in terms of performance, accessibility, best practices, SEO, and PWA rules. To do it, we need to integrate Lighthouse CI on our pipeline, to avoid deploying a new version of our web application if it doesn’t respect all quality rules.

Useful links

Thanks to Michał Czmiel and Żaneta Górska for helping me to implement Lighthouse CI on our product; Aurélien Lair, przemkow, Fabio Di Peri and Pasquale Mangialavori for the review of this article. Without you writing and publishing this article would not have been possible. 🙏