Generate End-to-End Testing reports using CodeceptJS, Playwright and Allure Report

Published in

Bear Necessities

6 min readOct 31, 2023

Reporting is crucial for end-to-end testing as it provides valuable insights into the test results and helps identify any issues or areas for improvement. With comprehensive reports, you can easily track the progress of your tests, analyse test outcomes, and identify patterns or trends in the test data. Reporting also enhances collaboration among team members by providing a clear overview of the testing process and facilitating effective communication. Additionally, reports can be used to showcase the quality and reliability of the application under test, making them essential for stakeholders and decision-makers. This article shows how we can achieve End-to-End test reporting with CodeceptJS, Playwright and Allure Report.

Playwright

Playwright is a robust testing framework that enables dependable end-to-end and cross-browser testing for web applications. It offers support for major web browsers like Chromium, Firefox, and WebKit. With Playwright, developers can utilise a unified API to interact with web browsers and write tests that can be executed across multiple browsers without the need for separate code for each browser.

CodeceptJS

CodeceptJS is a modern test automation framework that seamlessly integrates with different testing frameworks, such as Playwright, to simplify end-to-end testing for web, mobile, and desktop applications, including Electron.js. It provides a user-friendly syntax and a comprehensive range of features for effortlessly creating and executing automated tests.

Allure Report

Allure Report is a flexible and lightweight tool for generating and viewing clear graphical reports. It supports multiple languages and the majority of test frameworks. The Allure reports can be customised by adding additional metadata to the test cases, such as tags, labels, or descriptions. Integrating Allure Report with a CI/CD pipeline makes it useful for any automation process. With Allure Report, you can easily view and share test reports, enhancing collaboration and improving the overall testing process.

Install Allure Report

Install Allure Report executable to run locally on your development machine. The command below installs allure on macOS. Please follow the documentation to install on your operating system.

brew install allure

Ensure you have relatively latest version of Java SDK installed on your system and JAVA_HOME variable is configured properly as Allure Report requires Java runtime to run a local web server.

This will make allure executable accessible in your terminal. However, if you choose to install it using npm, it will be accessible within your project and you will need to create an npm script to run it. This is what most projects need.

npm install --save-dev allure-commandline

Install CodeceptJS Allure helper

Allure Report is integrated with CodeceptJS tests using an official plugin. Install allure-codeceptjs as an NPM dependency by running npm install command.

npm install --save-dev allure-codeceptjs

Modify your codecept.conf.ts configuration file to include Allure Report helper.

{
  plugins: {
    ...
    allure: {
      enabled: true,
      require: 'allure-codeceptjs',
      outputDir: './output/allure-raw',
    },
    ...
  }
}

As it is quite common to extend CodeceptJS using helpers as well, ensure that this configuration is added as a plugin and NOT a helper. Otherwise, the error below will be displayed.

Could not load helper allure from module 'allure-codeceptjs':
HelperClass is not a constructor
TypeError: HelperClass is not a constructor
    at createHelpers (/node_modules/codeceptjs/lib/container.js:192:29)
    at Function.create (/node_modules/codeceptjs/lib/container.js:44:25)
    ...

Generate reports

The report in HTML format can be generated by runningallure generate command.

allure generate ./output/allure-raw -o ./output/allure-report --clean

Use allure-patch, a handy utility from the community to combine Allure Report output results into a single HTML file.

allure-patch ./output/allure-report

View reports

Make sure tests are run and reports are generated. To view generated reports spin up the local web server and a new tab will automatically open browser window with interactive reports UI.

npx allure-commandline serve

The process needs to be restarted for new scenarios to be available in the reporting UI. If you have specified a custom location for the reports, use allure open command.

allure open ./output/allure-report

Deploy Allure Report to GitHub Actions pipeline

As now we can generate a static HTML bundle with a single HTML file, we can host it on the web, say using GitHub Pages. To achieve deployment to GitHub Pages, navigate to your GitHub repository Settings and click on Pages tab.

Select GitHub Actions as the Source in Build and deployment section.

Configure GitHub Actions pipeline

Create a new YAML pipeline file in .github/workflows folder or add a new job to your existing pipeline.

# Ensure the right permissions are set for GitHub Pages
permissions:
  contents: read
  pages: write
  id-token: write

jobs:
 # New Allure Report job to run tests and generate reports
 report:
    timeout-minutes: 30
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.38.1-jammy
    environment:
      # Exposing GitHub Pages url
      url: ${{ steps.deployment.outputs.page_url }}
      name: 'github-pages'

    steps:
      - uses: actions/checkout@v3
      
      - uses: actions/setup-node@v3
        with:
          node-version: 'latest'
      
      # Installing Java for Allure executable to run
      - uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '17'

      - name: Install dependencies
        run: npm ci

      - name: Run Chrome scenarios
        run: npm run test:chrome
        env:
          BASE_URL: ${{ vars.BASE_URL }}
          HEADLESS: ${{ vars.HEADLESS }}
          OUTPUT_PATH: ${{ vars.OUTPUT_PATH }}

      - name: Generate Allure report
        if: always()
        run: npm run report:generate

      - name: Patch Allure report
        if: always()
        run: npm run report:patch

      - name: Setup GitHub Pages
        if: always()
        uses: actions/configure-pages@v3

      - name: Upload an artifact for GitHub Pages
        uses: actions/upload-pages-artifact@v2
        if: always()
        with:
          path: ${{ vars.OUTPUT_PATH }}/allure-report

      - name: Deploy to GitHub Pages
        if: always()
        id: deployment
        uses: actions/deploy-pages@v2

Once we have GitHub Actions pipeline configured, executed and deployed, we can see the GitHub Pages url.

Please refer to CodeceptJS Playwright Typescript Boilerplate for the complete configuration for the pipeline.

Allure Report deployment to GitHub Pages

A sample Allure Report deployment is available in https://bear-plus.github.io/codeceptjs-playwright-typescript-boilerplate.

Keep historical data

We can enhance our Allure Report output with historical data that can be stored for each test scenario and later examined to give extra insights of how test scenarios are performing over time. To achieve this, we can use a custom GitHub Action called Allure Report with history.

Update your GitHub repository Settings by clicking on Pages tab and switch Source to Deploy from a branch.

Select gh-pages branch or create one if you do not have it already.

Configure your GitHub Actions job steps as described in the allure-report-action documentation.

- name: Get Allure Report history
  uses: actions/checkout@v2
  if: always()
  continue-on-error: true
  with:
    ref: gh-pages
    path: gh-pages

- name: Generate Allure Report
  uses: simple-elf/allure-report-action@master
  if: always()
  with:
    allure_results: ${{ vars.OUTPUT_PATH }}/allure-report
    allure_history: ${{ vars.OUTPUT_PATH }}/allure-history
    keep_reports: 5

- name: Deploy Allure Report to GitHub Pages
  if: always()
  uses: peaceiris/actions-gh-pages@v2
  env:
   PERSONAL_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    PUBLISH_BRANCH: gh-pages
    PUBLISH_DIR: ${{ vars.OUTPUT_PATH }}/allure-history

Summary

Comprehensive reports play a crucial role in end-to-end testing as they offer valuable insights into test results, enabling the identification of issues and areas for improvement. This article shows how we can achieve End-to-End test reporting with CodeceptJS, Playwright and Allure Report and deploy it to GitHub Pages.

Source code

The approach described in this article is implemented in CodeceptJS Playwright Typescript Boilerplate and available in Bear Plus GitHub space.