Scaling-out End-to-End Tests with 7Facette & Playwright to Infinity

by Patrick Döring

What is your goal for test automation? Is it to enable continuous testing as part of your built process? After checking in new features, your developers may want the confidence that what they have added is not compromising the overall quality of the application. Or is it to reduce the time that it takes to execute your automated tests? Perhaps your team needs to release features faster.

Being kind of a serverless enthusiast, I wanted to figure out if my tests can run in parallel without big code changes on AWS (Amazon Web Services) Lambda. In this blog post, I will walk you through how to do it step by step with 7Facette and Playwright.

In December 2020, Amazon launched a new feature that introduced container image support for Lambda functions — a game changer for running 7Facette & Playwright on AWS Lambda!

So, I started to build up a container image with all the dependencies and run some little tests deploying that as a Lambda function.

The function

I created a simple Java function that uses 7Facette to make an API call and Playwright to open the GitHub page of Playwright and print the page title to console.

To make this possible you must implement your own JUnit runner which calls your tests as an event.

The Docker Image

I started to build a docker image based on this post on the AWS blog. Let’s use “mcr.microsoft.com/playwright/java:focal” as the base image and create some directories.

The base image “mcr.microsoft.com/playwright/java:focal” installs browser executables under the home directory of a user called “pwuser” and creates symbolic links for other users. AWS Lambda uses its own pool of sandbox users called “sbx_userXXXX” that do not exist when the image is built. Thus, the users will not have symbolic links to the browser executables under their own home directories.

Next, we will copy the function directory — the actual Lambda function code — to the image and build it with Gradle. After that, we copy all generated artifacts to the “/opt/app” and the browser executables to the “/opt/app/ms-playwright/” folder.

Since we create an image using an alternative base image, we must set the “ENTRYPOINT” property to invoke the runtime interface client and also the “CMD” argument to specify the Lambda function handler.

Running your code — Locally with SAM

The AWS SAM command will build the docker image for us.

sam build

You can see the docker build steps in the console output. A genuinely nice aspect of the AWS SAM is the ability to run the API locally.

sam local start-api

Invoke the function with HTTP request. This is necessary because we expect a POST request including JSON body. This triggers the function, and we’ll get the output what we would see in the CloudWatch logs.

curl -XPOST "http://127.0.0.1:3000/test" -d '{"package": "de.aws.api", "class": "", "method": ""}'

Deploying to AWS Lambda

Finally, we can have AWS SAM create our Lambda and connect it to a REST API for us!

sam deploy --guided

Play by the rules in AWS Lambda

Everything seems to work fine on the local docker container but after deploying the arduous work to AWS Lambda you’ll notice how things are never so simple. Next, we will go through the issues I came across on the AWS Lambda environment and explain how they are solved.

No multiprocessing support

When running the function with Chrome, it fails at a process called “start_thread”. The function fails when Chromium is trying to spawn a new thread / process for something probably GPU (Graphics Processing Unit) related.

AWS Lambda runs in an environment that has several limitations. One of them is the lack of multiprocessing support. Modern browsers spawn multiple processes with a couple of different strategies like rendering. You can find more about Process Models for Chromium here.

Fortunately, with Chromium you can use a launch flag called “ — single-process” to disable the use of multiple processes.

When you disable the usage of multiple processes you also must disable the usage of zygote process with the flag “ — no-zygote”. A zygote process is one that listens for spawn requests from a main process and forks itself in response. Generally, they are used because forking a process after some expensively performed set-up can save time and share extra memory pages. Finally, you must disable the sandboxing for Chrome with “ — no-sandbox”.

Let’s have a look how to launch the browser with “args” option.

WebKit just works

WebKit browser seems to work out of the box. No issues at all! So far, there might be issues with some more specific use cases.

Firefox not running yet

Unfortunately, Firefox doesn’t have a similar launch argument to force single processing. Thus, I have not been able to run tests with Firefox.

Scale-out your tests without code changes

When you only have a handful of automated tests, running the sequentially doesn’t seem important. However, as the number of automated tests grows, the longer it will take for your entire test suite to complete. A solution to this is to run the tests in parallel to save time of the team. However, this is not as simple as flipping a switch, isn’t it?

Parallel testing with AWS Lambda

Scale-out is when multiple machines are configured to run tests in parallel. Whereas scale-up had one machine running multiple tests, scale-out has multiple machines each running tests.

Scale-out is a better long-term solution than scale-up because it can handle an unlimited number of machines for parallel testing. The limiting factor with scale-out is not the maximum capacity of the hardware but rather the cost of running more machines. However, with AWS Lambda scale-out is much easier to implement than scale-up.

The first time you invoke your function, AWS Lambda creates an instance of the function and runs its handler method to process the event. When the function returns a response, it stays active and waits to process additional events. If you invoke the function again while the first event is being processed, Lambda initializes another instance, and the function processes the two events concurrently. As more events come in, Lambda routes them to available instances and creates new instances as needed. When the number of requests decreases, Lambda stops unused instances to free up scaling capacity for other functions.

This has the advantage that we can run everything at once in parallel what makes our test execution as fast as possible without code changes. And we can have 1000 concurrency quotas with no additional costs and minimal maintenance.

Conclusion

In the end, I managed to get two out of three browsers working on AWS Lambda: WebKit and Chromium. It is of course better than nothing, but it seems that AWS Lambda is a tough environment to run modern web browsers.

Parallel testing is a worthwhile endeavor. When done properly, it will not only reduce development time but also improve the development experience. With AWS Lambda you don’t have to constantly observe servers or infrastructure. We can scale-out until 1000 concurrency quotas what makes our test execution as fast as possible without code changes, no additional costs and minimal maintenance.

All the code can be found on the GitHub repository. The first version (tagged: ‘v0.0.1’) is almost identical to the code examples shown on this blog post.