Before deploying the code in production, a best practice is running all required tests: unit, integration, and E2E tests. But why don’t you also test the performance and the accessibility of your web application?
In TUI Musement, every team implements unit and integration tests and we have a QA team dedicated to the automation of E2E tests, to run on a pre-production environment. So we have decided to add a quality test to maintain an excellent level of performance, accessibility, SEO and best practices for our products.
This guide will show you how to test the performance, accessibility, SEO, best practices, and PWA rules of your web application. In order to do this, we will add Lighthouse CI to a GitLab CI pipeline. Lighthouse CI uses Lighthouse, the tool integrated into Chrome-based browsers.
We will run the Lighthouse job on Merge Request creation and update and commit to the master branch, but you can change this easily, according to your needs.
Install Lighthouse CI
The first step is installing Lighthouse CI in your web app. You can install it globally or locally as a dev dependency. I prefer the second way:
npm install --save-dev @firstname.lastname@example.org
yarn add --dev @email@example.com
⚠️ I used version 0.7.2 of Lighthouse CI, so I am sure that this version works for all parts described below. But you can try the latest 0.x version, and it should also work.
Configure Lighthouse CI
The second step is configuring Lighthouse CI, to do this you need to create a new file called
lighthouserc.js in the root folder of your project.
👉 You can also configure Lighthouse CI using YAML or JSON, as described here.
The first section of the configuration file is relative to the
collect step: how Lighthouse runs the tests and collects the data. You can find detailed documentation here.
numberOfRuns indicates the number of runs that Lighthouse CI makes for each URL. Every run is independent of the others. The Lighthouse team suggests running it multiple times to reduce the variability of the results: a good value can be 5.
startServerCommand is the command to run the server before Lighthouse runs. In my repository, I used Nuxt and I needed to run the Nuxt server before, so I have used
npm start and it starts
url is the list of the URLs Lighthouse tests; each URL will be tested
If you don’t specify the property
onlyCategories, Lighthouse CI will test all categories (Performance, Accessibility, Best Practices, SEO and, Progressive Web App) on your web app, unless you specify exactly what categories you want to test. In this case, we are testing all categories except the PWA audits, because our web app will not be a Progressive Web Application:
You can use the property
chromeFlags to pass flags to the Chromium browser Lighthouse uses:
You can set extra headers or cookies using
extraHeaders: it can be one of the options to run Lighthouse on authenticated pages:
The configuration of the assertions is very important because a failure can stop your pipeline and, consequently, you cannot merge your branch or deploy your code to production if one or more assertions are not respected.
If you don’t need a custom configuration, you can use a preset (e.g.,
lighthouse:recommended ), otherwise, you can add custom assertions starting from a preset:
You can set an assertion on:
- single audits (e.g.,
- entire categories (e.g.,
For each assertion, you need to set:
- a level between
off(the audit is not checked),
warn(the audit is checked but the error generates only a warning, and the pipeline is not stopped) and
error(the audit is checked, the error returns a non-zero exit code, stopping the pipeline), as described here;
minScoreunder which the assertion fails;
medianRun, as described here.
In this case:
- the performance score must be greater or equal to 90 in the best run
- the accessibility score must be equal to 100 in all runs
- the best practices score must be equal to 100 in all runs
- the SEO score must be equal to 100 in all runs
The last section
upload indicates where Lighthouse saves the report data. You can find detailed documentation here.
You can choose to save them:
- on a temporary store with
target=temporary-public-storage. You can access the reports by clicking on the links at the end of the test
PROS: easiest and fastest way;
CONS: it’s public and temporary; no history;
- on a Lighthouse CI server with
PROS: historical archive of all tests;
CONS: time/resources needed to create a server, as described here and here;
- on the filesystem with
PROS: save (and download) reports as artifacts in the CI;
CONS: the access to the reports is less immediate than other methods; no history.
We proceed with the easiest way, for now, using a temporary store.
⚠️ The temporary store is public, so take care not to share sensitive data.
Run Lighthouse locally
At this time, you have configured Lighthouse and you can run it locally. You only need to add this new NPM task to your package.json:
Now you need to create your production bundle and run Lighthouse:
npm run build
npm run lighthouse// or with Yarn:
It prints something like this:
✅ .lighthouseci/ directory writable
✅ Configuration file found
✅ Chrome installation found
Healthcheck passed!Started a web server with "npm start"...
Running Lighthouse 5 time(s) on http://localhost:3000/
Running Lighthouse 5 time(s) on http://localhost:3000/detail/1
Done running Lighthouse!Checking assertions against 2 URL(s), 10 total run(s)All results processed!Uploading median LHR of http://localhost:3000/...success!
Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1621867616820-1661.report.html
Uploading median LHR of http://localhost:3000/detail/1...success!
Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1621867617506-21870.report.html
No GitHub repository slug found, skipping URL map upload.
No GitHub token set, skipping GitHub status check.Done running autorun.
- Lighthouse ran
npm startusing the configuration property
- when the server was ready, it ran the tests;
- in this case, all assertions were valid, unless it returned a non-zero exit code;
- it uploads the report on the temporary public store
Run Lighthouse on GitLab CI
It’s time to run Lighthouse on your GitLab CI, creating
.gitlab-ci.yml in the root folder.
cypress/browsers:nodeXX.XX.X-chromeXX-XX as Docker
image, as suggested in the official documentation of Lighthouse CI.
We defined only one stage called
test and it contains a job called
allow_failure: false, if it fails then the pipeline fails;
- using the property
script, it installs all dependencies (
npm install), creates the build (
npm run build), and runs lighthouse (
npm run lighthouse).
We defined a
workflow , as suggested here, to define how the GitLab CI pipeline works:
- it runs a merge request pipeline when a merge request is created/updated;
- it runs a branch pipeline when a commit is pushed on any branch, but there is not a merge request opened on that branch.
In this way, using the
rules property, the job
lighthouse runs on Merge Request creation/update or commit on the master branch:
Create a Merge Request
It’s time to test all creating a Merge Request.
We have a Nuxt application with 2 pages where we fetch a todos API. I created a Merge Request from a branch where I added a fake timeout of 5 seconds before fetch:
In this way, the Lighthouse will find a blank page for 5 seconds and we expect the performance score will be decreased.
As we set GitLab CI, the Lighthouse job ran on our Merge Request pipeline… and it fails:
We can discover why it’s failed, by opening the job in detail:
$ npm run lighthouse...Checking assertions against 2 URL(s), 10 total run(s)1 result(s) for http://localhost:3000/ :✘ categories.performance failure for minScore assertionexpected: >=0.9found: 0.83all values: 0.82, 0.83, 0.83, 0.81, 0.831 result(s) for http://localhost:3000/detail/1 :✘ categories.performance failure for minScore assertionexpected: >=0.9found: 0.86all values: 0.85, 0.85, 0.86, 0.85, 0.86Assertion failed. Exiting with status code 1.Uploading median LHR of http://localhost:3000/...success!Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1622222737726-80625.report.htmlUploading median LHR of http://localhost:3000/detail/1...success!Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1622222738383-49315.report.html...ERROR: Job failed: exit code 1
As you can see, there is a failure for 2 assertions: the performance score (0.83 and 0.86) is less than the minimum score (0.9) for both pages. It attaches the link to the Lighthouse report for all URLs analysed:
It exited with an error code, and the pipeline cannot continue. We cannot deploy our code to production until we have resolved all failed assertions.
Using Lighthouse CI we can maintain a good quality of our product, in terms of performance, accessibility, best practices, SEO, and PWA rules. To do it, we need to integrate Lighthouse CI on our pipeline, to avoid deploying a new version of our web application if it doesn’t respect all quality rules.
- the complete repository of this implementation
- the official guide to lighthouse-ci
- keyword reference for gitlab-ci.yml
- a useful guide on GitLab Performance with Lighthouse
Thanks to Michał Czmiel and Żaneta Górska for helping me to implement Lighthouse CI on our product; Aurélien Lair, przemkow, Fabio Di Peri and Pasquale Mangialavori for the review of this article. Without you writing and publishing this article would not have been possible. 🙏