Building an End-to-End Testing Pipeline with CodeceptJS, Playwright, and GitHub Actions

Photo by Jukan Tateisi on Unsplash

End-to-End Testing is a crucial aspect of software development as it ensures that all components of a system are functioning correctly together. CodeceptJS is an efficient and robust End-to-End automation framework with which you can avoid vendor lock-in and benefit from rich locators, interactive debugging, and much more. When combined with Playwright, it becomes a powerful tool for automating web, mobile, and even desktop (Electron.js) applications. In this article we will explore how to build a End-to-End testing pipeline with CodeceptJS, Playwright and GitHub Actions.

GitHub Actions

GitHub Actions is a powerful and flexible CI/CD platform that enables you to automate your software development workflows. With GitHub Actions, you can quickly automate your development, testing, or operational processes directly within your GitHub repository. GitHub Actions seamlessly integrates with CodeceptJS and Playwright, providing a reliable solution for your project’s test automation needs.

Prepare the environment

We will start with configuring the Environment Variables that will define how the pipeline will be running as well as control its behaviour. Typically projects run multiple environments, therefore we will define BASE_URL variable as a minimum to alternate between running environments.

Navigate to your GitHub repo settings and under Secrets and variables section select Actions. Clicking on Variables tab will show the list of all variables for this repo.

GitHub repository environment variables settings

Click on New repository variable to create BASE_URL that will point test automation to your web application url.

Other useful settings that can be implemented using environment variables:

• HEADLESS=true - to run headless mode inside the pipeline or headful on your local machine to debug test scenarios.
• DEVICE_SCALE_FACTOR=1 - avoid screen flickering while developing test scenarios locally (use 2 for MacBook screens).
• ACCESSIBILITY=true - include accessibility reports in your End-to-End Testing.
• RECORD_HAR=true - record network traffic in a HTTP archive to use for debugging purposes.

Configure the pipeline

Create a new file in YAML format to configure the pipeline and put it in .github/workflows folder.

// e2e-test-automation.yml
jobs:
  test-automation:
    timeout-minutes: 30
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.38.1-jammy

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v3
        with:
          node-version: 'latest'

      - name: Install dependencies
        run: npm ci

      - name: Run test automation
        run: npm run test
        env:
          BASE_URL: ${{ vars.BASE_URL }}
          HEADLESS: ${{ vars.HEADLESS }}
          OUTPUT_PATH: output

      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: artifacts
          path: output
          retention-days: 30

Multiple browsers

Define web browser profiles in CodeceptJS configuration file.

// codecept.conf.ts
{
 //...
 multiple: {
   desktop: {
       browsers: [
       { browser: 'chromium' },
       { browser: 'firefox' },
       { browser: 'webkit' }
     ]
   },
 },
 //...
}

Create separate npm scripts for each browser profile.

// package.json
"test:chrome": "npx codeceptjs run-multiple desktop:chromium --steps",
"test:firefox": "npx codeceptjs run-multiple desktop:firefox --steps",
"test:safari": "npx codeceptjs run-multiple desktop:webkit --steps",

This way of running CodeceptJS scenarios will provide detailed feedback on each step the test runner attempts to do.

Parallel test runs

To make the process more efficient, run test scenarios in parallel using multiple workers.

// package.json
"test:chrome": "npx codeceptjs run-workers 3 desktop:chromium",
"test:firefox": "npx codeceptjs run-workers 3 desktop:firefox --steps",
"test:safari": "npx codeceptjs run-workers 3 desktop:webkit --steps"

CodeceptJS will show detailed steps only for failed scenarios and this is what is needed in most cases.

Use consistent helper dependencies

We are using Playwright as a CodeceptJS helper in this case, so make sure Playwright npm package version matches the Docker image version in the pipeline.

// package.json
"devDependencies": {
  ...
  "codeceptjs": "^3.5.5",
  "playwright": "^1.38.1",
  ...
}

Double check the Docker image version in your GitHub Actions pipeline configuration.

// e2e-test-automation.yml
jobs:
  test-automation:
    timeout-minutes: 30
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.38.1-jammy

In case versions are inconsistent, the pipeline will fail with a message that is not too obvious.

GitHub Actions pipeline log with Playwright error message

Each browser gets a separate job

Run each browser on a separate GitHub Actions job to test your application in all browsers efficiently.

// e2e-test-automation.yml
jobs:
  chrome:
    timeout-minutes: 30
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.38.1-jammy

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v3
        with:
          node-version: 'latest'

      - name: Install dependencies
        run: npm ci

      - name: Run Google Chrome tests
        run: npm run test:chrome
        env:
          BASE_URL: ${{ vars.BASE_URL }}
          HEADLESS: ${{ vars.HEADLESS }}
          OUTPUT_PATH: output-chrome

      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: chrome-artifacts
          path: output-chrome
          retention-days: 30

  firefox:
    ...

  safari:
    ...

Fix Firefox runs on GitHub Actions

Running Playwright automation in Github Actions fails for Firefox with the error message: Running Nightly as root in a regular user’s session is not supported.

GitHub Actions pipeline log with Running Nightly as root in a regular user’s session is not supported error

To fix this issue we need to workaround the problem by adding a HOME environment variable into the pipeline.

// e2e-test-automation.yml
- name: Run Firefox scenarios
  run: npm run test:firefox
  env:
    BASE_URL: ${{ vars.BASE_URL }}
    HEADLESS: ${{ vars.HEADLESS }}
    OUTPUT_PATH: output-firefox
->  HOME: /root

More details could be found in the appropriate Playwright GitHub issue.

Complete GitHub Actions pipeline

// e2e-test-automation.yml
name: E2E Test Automation

on: [ push ]

jobs:
  chrome:
    timeout-minutes: 30
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.38.1-jammy

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v3
        with:
          node-version: 'latest'

      - name: Install dependencies
        run: npm ci

      - name: Run Chrome scenarios
        run: npm run test:chrome
        env:
          BASE_URL: ${{ vars.BASE_URL }}
          HEADLESS: ${{ vars.HEADLESS }}
          OUTPUT_PATH: output-chrome

      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: chrome-artifacts
          path: output-chrome
          retention-days: 30

  firefox:
    timeout-minutes: 30
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.38.1-jammy

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v3
        with:
          node-version: 'latest'

      - name: Install dependencies
        run: npm ci

      - name: Run Firefox scenarios
        run: npm run test:firefox
        env:
          BASE_URL: ${{ vars.BASE_URL }}
          HEADLESS: ${{ vars.HEADLESS }}
          OUTPUT_PATH: output-firefox
          # Workaround to fix GitHub Actions issue:
          # Running Nightly as root in a regular user's session is not supported.
          # See https://github.com/microsoft/playwright/issues/6500
          HOME: /root

      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: firefox-artifacts
          path: output-firefox
          retention-days: 30

  safari:
    timeout-minutes: 30
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.38.1-jammy

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v3
        with:
          node-version: 'latest'

      - name: Install dependencies
        run: npm ci

      - name: Run Safari scenarios
        run: npm run test:safari
        env:
          BASE_URL: ${{ vars.BASE_URL }}
          HEADLESS: ${{ vars.HEADLESS }}
          OUTPUT_PATH: output-safari

      - uses: actions/upload-artifact@v3
        if: always()
        with:
          name: safari-artifacts
          path: output-safari
          retention-days: 30

Summary

Building an end-to-end testing pipeline with CodeceptJS, Playwright, and GitHub Actions provides a powerful solution for automating your test scenarios. By leveraging the capabilities of CodeceptJS and Playwright, you can easily automate testing across different browsers and environments. GitHub Actions allows you to seamlessly integrate and automate your testing process within your GitHub repository. With the configuration and setup outlined in this article, you can ensure that your software development workflows are efficient and reliable, enabling you to deliver high-quality software with confidence.

Source code

The approach described in this article is implemented in CodeceptJS Playwright Typescript Boilerplate and available in Bear Plus GitHub space.