Running Selenium tests with Docker Compose

Running Selenium tests with Docker allows improving your build pipeline significantly.

It enables selenium tests execution in headless mode on every commit locally and remotely without extra setup and configuration. With only one simple command you will be able to run selenium tests and get test results. And it doesn’t matter where you run the tests on your local machine or they are executed on Jenkins server.

Most of the complexity in execution of the selenium tests on a local computer comes with installation overhead (for instance, various web browsers, web drivers like chromedriver or geckodriver, and possibly different versions). An execution of the selenium tests in Jenkins is complex also due to an additional reason. The tests have to be run headless, which requires additional setup and configuration of Jenkins server and its environment. Jenkins server is usually full of different stuff and it is an overhead at the initial stage as well as within a lifetime of the application under test.

Assume the following stack. One frontend application, git with Github are used as a version control system. Continuous Integration (CI) is done with the help of Jenkins Pipeline. The application under test is integrated with Docker. More than five frontend developers implement the application. The application has unit tests and integration selenium tests.

The goal is to simplify testing process, unify test execution environment, remove installation and configuration overhead and make the process faster and predictable.

Therefore, triggering a build on Jenkins should execute the selenium tests before new artefacts (for instance, a docker image) are pushed to a central repository. Additionally, a developer should be able to test a new version of the application under test even before pushing it outside her development environment.

To solve these challenges, selenium tests can be run using Docker. Docker removes all the complexity mentioned above in this article. However, before coming to the final solution and understanding its value, I will show first the basics of docker tool in the scope of the project available on Github. Docker can be used in various ways. Let me start.

Re-introducing the challenge

The goal is to run headless selenium tests for every possible version of the frontend application on a Jenkins server. To accomplish that with docker, I need all involved components to be integrated with docker. In this case, the components are:

  1. Project with selenium tests;
  2. Browser (e.g., chrome, firefox);
  3. Frontend application under test
  4. Jenkins

All these components have to be packed into docker images and be available locally or through a Docker Hub.

I will start with installing Docker.

Then, I need to integrate tests and application under tests with Docker. As for a headless browser, I will use selenium hub image from a public Selenium Docker Hub repository. Jenkins setup and installation is out of the scope of this article.

Make the test project ready for Docker

How do I do it? I introduce Dockerfile in the project with selenium tests (a directory with tests and other files). The source code of the sample project is on Github.

# Dockerfile
# Base image
FROM python:3.6
# Copy test project files to the image folder
COPY . /frontend-integration-tests
# Make the folder as a working directory
WORKDIR /frontend-integration-tests
# Install the test project libraries
RUN pip install -r requirements.txt

What does it mean? The docker image with the selenium tests will run on ubuntu with Python3.6 . All files in a test project will be copied to a directory called /frontend-integration-tests in the image and all dependencies specified in requirements.txt installed. The selenium tests is a Python project written using RobotFramework with SeleniumLibrary.

So after that’s ready, I can build a Docker image using a command

#sh
docker build .

or even better, I can give a name and a tag to the image with a docker command,

#sh
docker build -t robottests:1 .

where . is a location of the Dockerfile.

To check if the image is created run

#sh
docker images
REPOSITORY    TAG    IMAGE ID         CREATED           SIZE
robottests    1      ad4672c29046     1 minutes ago     745MB

When so, I can start a docker container and execute commands in it

#sh
docker run robottests:1 robot --variablefile variables/config.py --variable BROWSER:chrome tests/

where docker runis a docker command to start a docker container with robottests:1image and robot --variablefile variables/config.py -- variable BROWSER:chrome tests/ is a RobotFramework command to execute the tests.

The output is WebDriverException: Message: ‘chromedriver’ executable needs to be in PATH. ...

Selenium test fails when executed in a docker container

Why? The reason is that a test runner can’t find chrome web driver. Therefore, I configure the test runner to connect to a remote chrome web driver through Selenium Grid. Remember, I need headless way to execute the tests on Jenkins.

With Robot Framework SeleniumLibrary, I would start a web browser like this

# my_test.robot
Open Browser    {APP_URL}    {BROWSER}  remote_url={SELENIUM_GRID}

I know APP_UR = localhost:8080and BROWSER=chrome but an address of remote webdriver I do not know yet. To find it out, I use docker to standalone chrome driver with Selenium Grid

#sh
docker run -d -P selenium/standalone-chrome
CONT.. ID   IMAGE   COMMAND  CREATED    STATUS      PORTS   NAMES
1fb39a9     selenium/standalone-chrome   "/opt/bin/entry_po..."   13 seconds ago      Up 12 seconds  0.0.0.0:32873->4444/tcp wizard_sino

You do not have to build selenium/standalone-chrome image because it is available on Docker Hub selenium public repo.

Now I try http://0.0.0.0:32873/ and see that standalone chrome is up and running and I can use this address to configure the tests

#something.robot
Open Browser http://localhost:8080 Chrome remote_url= http://localhost:32873/

I build an image with the selenium tests again because I modified them by adding theremote_url and then I run the tests in a new container with a new image

#sh
docker build -t robottests:2 .
#sh
docker run robottests:2 robot --variablefile variables/config.py --variable BROWSER:chrome tests/

In result, the test are executed, and they Pass.

Selenium test pass when executed in a docker container

This was a step-by-step instructions about running selenium tests with Docker. However, it becomes overwhelming to build-run these commands every time you need to execute the selenium tests.

Usually, if you have more than one docker container setup you have to turn it into scalable solution and that’s where Docker Compose comes in handy.

Docker Compose

First, I create a docker-compose.yaml where I specify services I want to run and how to find them: tests, application and web drivers. Also, I want to run the selenium tests in an isolated network and I want to copy a test report from the docker container to my local file system where I can actually access them. In a docker terminology, I want to use user defined network and mount a volume into a container with the test runner.

Illustration. Docker containers in a user defined network.

Here is docker-compose.yaml and here is a Github repo with a small project I use to explain the solution.

{WORKSPACE} is a Jenkins environment variable that defines a root of a repository with the project files. Volume allows to specify a location to which a test report will be copied, otherwise, the test reports after execution will stay inside the container.

robottestnw is a user defined network . The network allows to isolate the containers in a sub-network as well as find each other in it.

robottests, app or chromenode are aliases of the services which can be used in docker context instead of IP addresses.

Because container with the tests has to start at the end and wait for another container to start, it’s necessary to specify an order with depends_on parameter. The selenium tests depend on the app and selenium grid.

Also, testui/wait-for-it.sh script waits for chromenode container to be up.

With a docker-compose, when the setup is ready we need to execute only one command

#sh
docker-compose up --build
A fragment of a test execution with docker-compose

or to run another command inside a specific container from the docker-compose.yaml

docker-compose run robottests robot -d reports  --variablefile variables/config.py  --variable BROWSER:chrome tests/
A test execution with docker-compose

run command in this case will not build the images. To build a specific image run

docker-compose build robottests

After the execution, it’s possible to stop the running containers and delete them with a command

docker-compose down

Integration the Selenium Tests into Jenkins Pipeline

Jenkins Pipeline (or simply “Pipeline” with a capital “P”) is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. — https://jenkins.io/doc/book/pipeline/

The definition of a Jenkins Pipeline is written into a text file called a Jenkinsfile and checked in to source control together with a relevant project. Jenkinsfile looks the following way

I use {BUILD_NUMBER}, which is a Jenkins environment variable, to tag the application under tests image.

The triggered Pipeline checks out a project from the source control, builds images with an app under tests and test project respectively, executes the tests with the docker-compose and stops all docker containers.

UPD. Recommendations

Usually, you will need to run the setup for a few projects on the same Jenkins server. It leads to conflicting situations between docker containers. Thus it’s important to isolate execution of these projects from each other. In docker compose file make sure you:

  1. Give a name to every container in the docker compose file:
robottests:
container_name: dashboard_robottests
command: /bin/sleep infinity

It helps to recognise your containers when you execute docker ps command and you will be able to clean environment by stoping or removing the containers.

2. Specify a project name

If the project name is specified, it’s possible to run several the same docker-compose setups on the same environment. You can imagine a situation when a few pull requests are executing Selenium tests in parallel. To avoid a conflict between the executions run the tests using the following command:

docker-compose -p unique_name up

3. Wait when Selenium Grid and Chrome Standalone are ready

Selenium Grid has an internal state. This state should beready: truebefore the Selenium Grid can create a new webdriver session.

The status is available at http://chromenode:5555/status or http://selenium_hub:4444/status. Before the test runner starts the tests, it better check if the status of the Selenium Grid, as well as all connected web drivers, is ready. Otherwise, the test runner will not be able to create a web driver session and the tests will fail.

Here is an example how this check in form of a Smoke Test could look like.

Conclusion

With a simple configuration set up only once, it’s possible to run headless selenium tests in any environment (local computer or Jenkins server) with only one command

docker-compose up --build