Running Jest programmatically with Puppeteer in AWS Lambda and CDK

8 min readJun 17, 2024

A couple of weeks ago, I stumbled upon a scenario. I wanted to execute jest test cases with Puppeteer, programmatically in AWS Lambda.

If you are in a hurry feel free to skip everything and scroll to the last part for quick setup steps.

For the rest, let's start.

Building blocks

The main frameworks and infra in this scenario are —

Jest

Jest is a JavaScript testing framework designed to ensure correctness of any JavaScript codebase. It allows you to…

jestjs.io

Puppeteer | Puppeteer

build

pptr.dev

What is the AWS CDK?

The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework for defining cloud…

docs.aws.amazon.com

Managing Lambda dependencies with layers

This section describes how to use AWS layers to separate your application code from its dependencies.

docs.aws.amazon.com

The sequence of challenges

The first challenge came with Puppeteer. puppeteer package comes with chromium installed by default which will not work in lambda due to package size constraints. The solution — to install puppeteer-core which doesn't include chromium and to install a trimmed version of chromium as a lambda layer

@sparticuz/chromium was the saviour .

Creating the puppeteer layer

using @sparticuz/chromium was easy and straightforward

Clone the repo
git clone — depth=1 https://github.com/sparticuz/chromium.git
Run Make
cd chromium && make chromium.zip
Create a Lambda layer and upload the chromium.zip . I have used nodejs16.x as the run time and x86_64 architecture

I tested the above steps by running a sample code.

const puppeteer = require("puppeteer-core");
const chromium = require("@sparticuz/chromium");
exports.handler = async (event, context) => {
  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath(
    process.env.AWS_EXECUTION_ENV
    ? '/opt/nodejs/node_modules/@sparticuz/chromium/bin'
    : undefined,
      ),
    headless: chromium.headless,
    ignoreHTTPSErrors: true,
  });
  const page = await browser.newPage();
  try {
    const domain = new URL("https://www.google.com").host;
    await page.goto(urlToRead, { timeout: 60000, waitUntil: ['load', 'networkidle0', 'domcontentloaded'] });
    const pageTitle = await page.title();
    console.log(pageTitle)
  } catch (error) {
    console.log(error);
  } finally {
    await page.close();
    await browser.close();
  }
    return;
};

The puppeteer seemed to be working well, and the next steps seemed easy. It was just running the jest programmatically, but boy I was wrong!!

This was the second challenge, I was using AWS CDK with typescript for creating this lambda, There were a lot of missing babel dependencies errors coming from internal jest packages when building the lambda, it was something to do with the docker image the CDK was using. The best solution was again lambda layers. I built a simple lambda layer for jest.

Creating The Jest layer

Create a directory and install Jest and dependencies
mkdir jest-layer/nodejs cd jest-layer/nodejs npm i jest jest-environment-node
Zip the files, create the layer, and upload the zip file with the same setting as the previous puppeteer layer

This was a bit easy, (don't forget to add jest as external dependency in the cdk) but …voila!! there comes the next one……

The Setup Problem: To use jest with puppeteer there was 2 options easiest being to usejest-puppeteer but this is not an option in my case, as the jest-puppeteer requires puppeteer package, but as you remember we are using puppeteer-core . So I used the manual setup as explained here

Using with puppeteer · Jest

With the Global Setup/Teardown and Async Test Environment APIs, Jest can work smoothly with puppeteer.

jestjs.io

the only change from the above document is the file setup.js .Here I had to use puppeteer-core and @sparticuz/chromium . The changed file is

// setup.js
const {mkdir, writeFile} = require('fs').promises;
const os = require('os');
const path = require('path');
const puppeteer = require('puppeteer-core');
const chromium = require("@sparticuz/chromium");

const DIR = path.join(os.tmpdir(), 'jest_puppeteer_global_setup');

module.exports = async function () {
  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath(),
    headless: chromium.headless,
  });
  // store the browser instance so we can teardown it later
  // this global is only available in the teardown but not in TestEnvironments
  globalThis.__BROWSER_GLOBAL__ = browser;

  // use the file system to expose the wsEndpoint for TestEnvironments
  await mkdir(DIR, {recursive: true});
  await writeFile(path.join(DIR, 'wsEndpoint'), browser.wsEndpoint());
};

jest.config.js will be

module.exports = {
    globalSetup: './setup.js',
    globalTeardown: './teardown.js',
    testEnvironment: './puppeteer_environment.js',
    testTimeout: 20000,
  };

So far so good, now I just need to upload the jest config and other associated files mentioned in the jest puppeteer documentation, but how do I do it using CDK?

The Easiest Challenge in the sequence

export class JestRunnerLambda extends NodejsFunction {
  constructor(scope: Construct, name: string, props) {
    super(scope, name, {
      entry: `lib/handlers/jest-runner.ts`,
      functionName: name,
      memorySize: 512,
      timeout: Duration.seconds(120),
      runtime: Runtime.NODEJS_16_X,
      bundling: {
        externalModules:["@sparticuz/chromium","jest"],
        commandHooks: {
            afterBundling: (inputDir: string, outputDir: string): string[] => [
              `cp ${inputDir}/lib/support-scripts/jest/jest.config.js ${outputDir}`,
              `cp ${inputDir}/lib/support-scripts/jest/puppeteer_environment.js ${outputDir}`,
              `cp ${inputDir}/lib/support-scripts/jest/setup.js ${outputDir}`,
              `cp ${inputDir}/lib/support-scripts/jest/teardown.js ${outputDir}`,

            ],
            beforeBundling: (inputDir: string, outputDir: string): string[] => [],
            beforeInstall: (inputDir: string, outputDir: string): string[] => [],
          },
      },
      layers: [
        LayerVersion.fromLayerVersionArn(
          scope,
          "chromium-cdk",
          "arn:aws:lambda:ap-south-1:404771918283:layer:chromium:1"
        ),
        LayerVersion.fromLayerVersionArn(
          scope,
          "jest-cdk",
          "arn:aws:lambda:ap-south-1:404771918283:layer:jest:1"
        ),
      ],
      role: your_role
    });
  }
}

So till now it was all about the infra part and getting the building blocks up and ready. Now is the final and most important part!

Running jest programmatically.

To run jest programmatically I used runCLI function since it supports dynamic configs. You can also use run if you are not planning to change the jest config in run time.

The challenge here was that I couldn't find any way to pass the test scripts dynamically, the runCLI method scans the project root path for test files and executes them. I am sure there might be some way to pass the script as a string, but due to time constraints, I chose to use the easiest option, i.e.

Get the test code from DB

2. Write it to a file, and execute the test.

Lambda allows writes only to the /tmp folder so I used /tmp folder as my project root

The handler function would be like this

import { runCLI } from "jest";
import { writeFile } from "node:fs/promises";
const jestConfig = {
  testTimeout: 10000,
};
export async function handler(
  event: APIGatewayEvent
): Promise<APIGatewayProxyResult> {
  try {
    const content = getCodeFromDB(testCase) //returns test scripts as string
    await writeFile("/tmp/dummy.test.js", content);
    const result = await runCLI(jestConfig as any, ["/tmp"]);
    if (result.results.success) {
      console.log(`Tests completed`);
    } else {
      console.error(`Tests failed`);
    }
    return result.results.success;
  }
  catch(e){
    console.log(e)
    return e
  }
}

The Final Challenge was to use one directory for storing test scripts and one directory for storing configs. Jest expects the config to be in the project root and the test scripts in the same or child folders.

Since we can keep the test script only in /tmp the folder which exists only in the lambda run time and we are bundling the configs at the build time which uploads it to /var the only way I could get this whole setup to work was ….

To read configs from /var and write it to /tmp when lambda is triggered.

Not the best solution, but it does the job well. I could have made it a bit more efficient by avoiding the reading part by having the configs in memory as strings, but it was not a maintainable option. The final handler looks like this

import { runCLI } from "jest";
import { writeFile, readFile } from "node:fs/promises";
const jestConfig = {
  testTimeout: 10000,
};
const configFiles = ["setup.js","teardown.js","puppeteer_environment.js","jest.config.js"]
export async function handler(
  event: APIGatewayEvent
): Promise<APIGatewayProxyResult> {
  try {
    // move the config files to /tmp
    await Promise.all(configFiles.map(file=>{
      const fileContent = await readFile(`/var/task/${fileName}`, "utf-8");
      await writeFile(`/tmp/${fileName}`, fileContent);
      return;
   }))
    const content = getCodeFromDB(testCase) //returns test scripts as string
    await writeFile("/tmp/dummy.test.js", content);
    const result = await runCLI(jestConfig as any, ["/tmp"]);
    if (result.results.success) {
      console.log(`Tests completed`);
    } else {
      console.error(`Tests failed`);
    }
    return result.results.success;
  }
  catch(e){
    console.log(e)
    return e
  }
}

That was all it phew !!. Finally, the lambda was running as I wanted.

Oh! Almost forgot, there was one last challenge which probably was the most difficult one for me which was —

Writing about it here ;). Writing is not my strong point, hopefully, it gets better as I go …

Quick Steps

1.Create the Lambda Layer using @sparticuz/chromium

Clone the repo
git clone — depth=1 https://github.com/sparticuz/chromium.git
Run Make
cd chromium && make chromium.zip
Create a Lambda layer and upload the chromium.zip

2. Create the Jest layer

Create a directory and install Jest and dependencies
mkdir jest-layer/nodejs cd jest-layer/nodejs npm i jest jest-environment-node
Zip the files, create the layer, and upload the zip file

3. Setup jest-puppeteer

setup.js

const {mkdir, writeFile} = require('fs').promises;
const os = require('os');
const path = require('path');
const puppeteer = require('puppeteer-core');
const chromium = require("@sparticuz/chromium");

const DIR = path.join(os.tmpdir(), 'jest_puppeteer_global_setup');

module.exports = async function () {
  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath(),
    headless: chromium.headless,
  });
  // store the browser instance so we can teardown it later
  // this global is only available in the teardown but not in TestEnvironments
  globalThis.__BROWSER_GLOBAL__ = browser;

  // use the file system to expose the wsEndpoint for TestEnvironments
  await mkdir(DIR, {recursive: true});
  await writeFile(path.join(DIR, 'wsEndpoint'), browser.wsEndpoint());
};

puppeteer_environment.js

const {readFile} = require('fs').promises;
const os = require('os');
const path = require('path');
const puppeteer = require('puppeteer-core');
const NodeEnvironment = require('jest-environment-node').TestEnvironment;

const DIR = path.join(os.tmpdir(), 'jest_puppeteer_global_setup');

class PuppeteerEnvironment extends NodeEnvironment {
  constructor(config) {
    super(config);
  }

  async setup() {
    await super.setup();
    // get the wsEndpoint
    const wsEndpoint = await readFile(path.join(DIR, 'wsEndpoint'), 'utf8');
    if (!wsEndpoint) {
      throw new Error('wsEndpoint not found');
    }

    // connect to puppeteer
    this.global.__BROWSER_GLOBAL__ = await puppeteer.connect({
      browserWSEndpoint: wsEndpoint,
    });
  }

  async teardown() {
    if (this.global.__BROWSER_GLOBAL__) {
      this.global.__BROWSER_GLOBAL__.disconnect();
    }
    await super.teardown();
  }

  getVmContext() {
    return super.getVmContext();
  }
}

module.exports = PuppeteerEnvironment;

teardown.js

const fs = require('fs').promises;
const os = require('os');
const path = require('path');

const DIR = path.join(os.tmpdir(), 'jest_puppeteer_global_setup');
module.exports = async function () {
  // close the browser instance
  await globalThis.__BROWSER_GLOBAL__.close();

  // clean-up the wsEndpoint file
  await fs.rm(DIR, {recursive: true, force: true});
};

jest.config.js

module.exports = {
    globalSetup: './setup.js',
    globalTeardown: './teardown.js',
    testEnvironment: './puppeteer_environment.js',
    testTimeout: 20000,
  };

4. Create Lambda using CDK and bundling the config files

export class JestRunnerLambda extends NodejsFunction {
  constructor(scope: Construct, name: string, props) {
    super(scope, name, {
      entry: `lib/handlers/jest-runner.ts`,
      functionName: name,
      memorySize: 512,
      timeout: Duration.seconds(120),
      runtime: Runtime.NODEJS_16_X,
      bundling: {
        externalModules:["@sparticuz/chromium","jest"],
        commandHooks: {
            afterBundling: (inputDir: string, outputDir: string): string[] => [
              `cp ${inputDir}/{pathto}/jest.config.js ${outputDir}`,
              `cp ${inputDir}/{pathto}/puppeteer_environment.js ${outputDir}`,
              `cp ${inputDir}/{pathto}/setup.js ${outputDir}`,
              `cp ${inputDir}/{pathto}/teardown.js ${outputDir}`,

            ],
            beforeBundling: (inputDir: string, outputDir: string): string[] => [],
            beforeInstall: (inputDir: string, outputDir: string): string[] => [],
          },
      },
      layers: [
        LayerVersion.fromLayerVersionArn(
          scope,
          "chromium",
          "your_layer:arn"
        ),
        LayerVersion.fromLayerVersionArn(
          scope,
          "jest",
          "your_layer:arn"
        ),
      ],
      role: your_role // replace with a Role
    });
  }
}

5. Creating the lambda handler function

import { runCLI } from "jest";
import { writeFile, readFile } from "node:fs/promises";
const jestConfig = {
  testTimeout: 10000,
};
const configFiles = ["setup.js","teardown.js","puppeteer_environment.js","jest.config.js"]
export async function handler(
  event: APIGatewayEvent
): Promise<APIGatewayProxyResult> {
  try {
    // move the config files to /tmp
    await Promise.all(configFiles.map(file=>{
      const fileContent = await readFile(`/var/task/${fileName}`, "utf-8");
      await writeFile(`/tmp/${fileName}`, fileContent);
      return;
   }))
    const content = getCodeFromDB(testCase) //returns test scripts as string
    await writeFile("/tmp/dummy.test.js", content);
    const result = await runCLI(jestConfig as any, ["/tmp"]);
    if (result.results.success) {
      console.log(`Tests completed`);
    } else {
      console.error(`Tests failed`);
    }
    return result.results.success;
  }
  catch(e){
    console.log(e)
    return e
  }
}