Web Performance: Integrating Lightouse CI in your Gitlab CI/CD pipeline

Since a long time, like any mature team, we have wanted to setup a performance measurement pipeline for our project. Some of the existing tooling that we explored related to this were webpagetest, lighthouse-ci, pagespeed insights API.

We figured out that webpagetest has an api, and provides cloud provider specific virtual machine images to setup your own private instances for server and client, where the server can orchestrate clients’ lifecycle and run tests on them. It also provides a public api which can be used with a key to get performance stats for your website from public servers of webpagetest and use that information to visualize the performance stats.

Pagespeed insights has an api, which is free to use, provides lab data as well as field data, and which can be used directly to plot charts in Graphana.

lighthouse-ci can be integrated in your ci pipeline and can help you see regressions in your performance scores with events (push, PRs) on your git repository.

Which workflow is better and what we would end up using is a discussion for another day. In this article, we would go through the series of steps which can be followed to setup lighthouse-ci as part of your gitlab CI/CD. This is more intended as a proof of concept rather than a full fledged solution. This might not be very beginner friendly, and it is assumed that due research is done, or can be figured out from the inline and reference links as mentioned in this doc.

Here is a list of steps for setting up lighthouse-ci with Gitlab CI/CD.

Adding gitlab-ci.yml

If this is the first time you are trying to setup a CI/CD pipeline for a project hosted on gitlab, the first thing that you would need to do is to create a gitlab-ci.yml file at the root of your project. This file, in its most basic usage, contains a list of jobs to be run as part of your CI pipeline in gitlab. By default, Gitlab would run all the jobs specified here with every push to the repository. You might want to configure it to only run on certain events, say, Merge Requests.

More details here:
https://docs.gitlab.com/ee/ci/yaml/
https://docs.gitlab.com/ee/ci/merge_request_pipelines/

We want to run lighthouse-ci as part of our gitlab pipeline. This is the set of steps as mentioned on lighthouse-ci Getting Started for gitlab:

image: cypress/browsers:node10.16.0-chrome77
lhci:
  script:
    - npm install
    - npm run build
    - npm install -g @lhci/cli@0.3.x
    - lhci autorun --upload.target=temporary-public-storage --collect.settings.chromeFlags="--no-sandbox" || echo "LHCI failed!"

This is a very basic config, and is suitable for static apps. This config would autorun the lhci with some sensible defaults and upload the lighthouse results to a temporary storage on cloud.More details available in the Getting started guide.

Later, we will see how to tweak this config so that it is suitable for our use case, an app which requires a server.

Gitlab Runners

Gitlab runners are virtual machines which run the jobs as specified in gitlab-ci.yml. There are some default public shared runners available in gitlab which you can use for any experimentation and may be for some actual jobs as well, but more often than not, these default shared runners wouldn’t be enough for you as they are public and live on the internet, and if your project setup is behind a VPN, you would need a runner which lives inside your VPN, or has access to the required set of things within your VPN.

https://docs.gitlab.com/ee/ci/runners/README.html

We will setup a gitlab runner on our local machine (MacOS):

Installing the gitlab-runner service:

• brew install gitlab-runner. https://docs.gitlab.com/runner/install/osx.html

Register a runner:

NOTE: I needed to register a runner as sudo and run all commands as sudo, otherwise the runner wasn’t visible on my gitlab CI/CD.

• register the runner using sudo gitlab-runner register. This will prompt you for various inputs.
• Enter your gitlab instance url, in my case it was https://gitlab.com
• Enter your project specific gitlab token. This token can be obtained from Settings -> CI/CD -> Runners for the specific project
• Enter a description for the runner (This can be changed later from gitlab UI)
• Enter relevant tags. This runner would pick up jobs which are tagged with the tags this runner is registered with.
• Enter the executor as Docker, as the job scripts we have specified for lightouse-ci run inside a docker container.
• Enter a default image which would be used if no image is specified in the job configuration. For now, you can just specify the exact same image as mentioned in the lighthouse-ci job configuration.

Now the runner should be registered successfully. We need to start this runner. Once started, the runner’s status should be updated to green in Gitlab.

• Run the multi runner service on your local machine. sudo gitlab-runner run
• from the Gitlab UI CI/CD settings, edit the runner properties to allow it to run untagged jobs as our current job configuration for lighthouse-ci does not specify any tags, and by default, runners only pickup tagged jobs, so this job wouldn’t be picked up by our registered runner by default.

TROUBLESHOOT: when running a job, make sure that the service is in running state, otherwise the job would remain in pending state.

As mentioned before, the lhci job requires docker to run a container with the specified docker image, we need to install docker on our system.
https://hub.docker.com/editions/community/docker-ce-desktop-mac

Lighthouse CI job configuration

Let’s breakdown the configuration as mentioned in lighthouse docs for a static site:

image: cypress/browsers:node10.16.0-chrome77
lhci:
  script:
    - npm install
    - npm run build
    - npm install -g @lhci/cli@0.3.x
    - lhci autorun --upload.target=temporary-public-storage --collect.settings.chromeFlags="--no-sandbox" || echo "LHCI failed!"

• image specifies the docker image to use. It contains cypress which would help run an instance of chrome on the runner, which would then be used to run lighthouse and collect scores.
• lhci is the name of the ci job. the script section defines the sequence of steps that should be run.
• npm install all the dependencies. Later we will see that it would be better to replace this with npm ci
• run the build command to build all static files in your project.
• install @lhci/cli globally on the runner virtual machine
• autorun lhci with sensible defaults, without a lighthouserc.json, and overriding some parameters.
• specify upload.target as temporary public storage, which is a link where your results are kept for 7 days temporarily. This is better explained in the lighthouse readme https://github.com/GoogleChrome/lighthouse-ci/blob/master/docs/getting-started.md#collect-lighthouse-results

Things we need to change here:

• First off, we would need to customize and override a lot more flags, so it would make sense to keep the lhci configuration in a separate file, lighthouserc.json.
• our project is SSR, so it requires a server to be run. This section in the doc helps us getting started with the setup, though later we would see the final configuration which required a little bit more tweaking, specifically, we need a way for lighthouse to know when our server is ready, for which we use “startServerReadyPattern”
• We don’t want to upload our results to a temporary public storage, instead we would like to upload our scores and analysis to another server, where we can use this information for visualizations. Fortunately, lighthouse also provides that setup in the form of LHCI server. https://github.com/GoogleChrome/lighthouse-ci/blob/master/docs/server.md

lighthouserc.json

Important steps that we want to override in our configuration are collect, assert and upload.

1. We need to tell lighthouse how to collect the scores.

For this, we need to tell lighthouse-ci how it can run lighthouse on our files. In this case, we have a SSR setup, so we need to tell lighthouse how to run our server, how to know when the server is ready to accept requests, and what is the server url. The final config would look something like this:

"collect": {
    "startServerCommand": "ENV=production npm run start",
    "url": "http://localhost:8083",
    "startServerReadyPattern": "my project running on 8083",    "settings" : {
    "chromeFlags": "--no-sandbox"
        }
},

where startServerReadyPattern is the message that the server logs on the console when it is ready to listen to requests.

2. We can add some custom rules to assert, to specify when the job should fail. We wouldn’t do that for now as it is a rather involved step. https://github.com/GoogleChrome/lighthouse-ci/blob/master/docs/getting-started.md#overview

3. We need to tell lighthouse that we want it to upload the results to a lighthouse server.

"upload": {
    "target": "lhci",
    "serverBaseUrl": "http://localhost:9001",
    "token": "token from the lighthouse server"
}

In a further section, we would see how to get this token. Also note that we are using localhost urls for the app server as well as the lighthouse server, as for now, both of them would be run on the same machine. (The gitlab runner and the lighthouse server would be installed on our local machine). Since the runner would be run in a container on our local machine, this also means that the docker container should be able to access the url for lighthouse server, which in my case was enabled by default for docker desktop.

Integrating the lhci server and uploading the lighthouse stats to it for visualization

https://github.com/GoogleChrome/lighthouse-ci/blob/master/docs/getting-started.md#the-lighthouse-ci-server

• install @lhci/server in a newly created repo.npm install -D @lhci/cli @lhci/server sqlite3 . (This repo does not need to have any code as of now, so you may as well install and run the above command globally.)
• run it using npx
npx lhci server --storage.storageMethod=sql --storage.sqlDialect=sqlite --storage.sqlDatabasePath=./db.sql

Now use the lhci wizard command to register a new project:

• ? Which wizard do you want to run? new-project
• ? What is the URL of your LHCI server? http://localhost:9001
• ? What would you like to name the project? my_test
• ? Where is the project’s code hosted https://github.com/<org>/<repo>

That’s it. You would get a build token and an admin token. You should use the build token in the upload section of your lighthouserc.json. Push to your repo, it would automatically run the job on the runner on your local, and upload the results to lighthouse server running on your local. Go to localhost:9001 and you would see something like this:

Lighthouse server UI

Final gitlab-ci.yml

image: cypress/browsers:node10.16.0-chrome77
lhci:
  script:
    - npm install -g npm
    - npm ci
    - ENV=production npm run build
    - npm install -g @lhci/cli@0.3.x
    - lhci autorun || echo “LHCI failed!”

Note that we are installing npm in the first step (see troubleshooting section for info), and using npm ci as opposed to npm install to install our dependencies (see references) . The autorun command takes its parameters from the lighthouserc.json .

Next steps:

• integrate with Merge requests against development branch.
• Figure out where this sits in our performance pipeline. This would help with running regression tests with every Merge Request, but may be we should still use Pagespeed insights api to get Field Data for our webapp.
• Figure out how the dashboards can be made better and specific to our use case.
• Figure out project specific assertions which should fail the job.

Troubleshooting:

• if you face issues with npm install, specifically with a package not being found, check if you have any dependencies which require a proxy. In my case, it was a package from a private npm registry.
• in the job script, just install latest version of npm before running anything else npm related, otherwise you might face the cb() never called error in npm install.
  https://stackoverflow.com/a/52688621/2054671
• docker container should be able to access the url for lighthouse server.
• run gitlab-runner with sudo
• doesn’t hurt to run with chrome “ — no-sandbox” flag. https://github.com/GoogleChrome/lighthouse-ci/blob/master/docs/troubleshooting.md#lighthouse-is-failing-to-run-how-do-i-fix-it
• when running docker desktop on mac, you might run into a problem where docker does not respond when running from system tray: https://github.com/docker/for-mac/issues/4375

References/ More info:

• https://stackoverflow.com/questions/52499617/what-is-the-difference-between-npm-install-and-npm-ci
• https://github.com/GoogleChrome/lighthouse-ci/blob/master/docs/getting-started.md