Running a Selenium Grid using Docker and Compose
As an engineer who’s responsible for maintaining an automated testing system for continuous integration, building it and making it reliable can be a challenging and time-consuming venture — but one that ultimately pays off. Conductor Searchlight, Conductor’s content intelligence platform, has an ever-growing list of features and nuanced business rules, all of which we need to verify are free from regressions and behaving as expected. To help us accomplish this goal in a timely fashion, we employ a fleet of automated tests, running concurrently on a Selenium Grid.
Selenium Grid
Selenium Grid is a great way to speed up your tests by running them in parallel on multiple machines. However, rolling your own grid also means maintaining it. Setting up the right browser / OS combinations across many virtual machines (or — even worse — physical machines) and making sure each is running the Selenium Server correctly is a huge pain. Not to mention troubleshooting when something goes wrong on an individual node. God help you when it comes time to perform any updates.
Sure, you could always go with third-party solutions like SauceLabs or BrowserStack, but these can be expensive to justify.
Thinking about this problem, it’s clear we need a central point from where we can configure and update our hub / nodes. It would also be great if we had a way to quickly recover in case an individual node crashes or otherwise ends up in a bad state.
Docker Gets You More Bang for Your Buck
For the uninitiated, Docker is a lightweight container (read: a scaled-down VM) that provides a fast and programmatic way to run distributed applications. Selenium Grid is distributed system of nodes for running tests. Instead of running your grid across multiple machines or VMs, you can run them all on a single large machine using Docker. The two are practically destined for each other.
Docker Compose (Formerly Known as Fig)
Enter Docker Compose (formerly known as Fig). This is what lets us take our Docker images and spin them up as a pre-configured Selenium Grid cluster. Together, the three technologies work almost perfectly: Selenium Grid handles the routing of your tests to the proper place, Docker lets you configure your browser / OS combinations in a programmatic way, and Compose is the central point from which you can spin everything up on the fly.
Optimize Your Selenium Grid
A huge benefit of using Docker is its capacity to scale. Running Selenium Grid on separate machines or even a set of VMs requires a lot of unnecessary computing overhead. Docker images run as userspace processes on a shared OS; so your images share some system resources, but are still isolated and require far fewer resources to run than a VM. This means you can cram more nodes into a single instance. Docker and Compose also take care of the networking for you! And there’s one more thing…
Nuke it from orbit
In my experience, nodes in the Selenium Grid inevitably end up in a bad state if they’re up and running for a long enough time, and there’s little value in diagnosing the root cause — it’s just faster to start over with a blank slate. And lightweight containers mean your grid’s hub and nodes can start up in seconds, not minutes. So if one of your nodes ends up in a bad state, rather than figuring out which node is wedged (and digging through its logs looking for a clue) you can just restart the entire grid. With Docker and Compose, you’re back up and running in less than 5 minutes.
How to Run Your Selenium Grid with Docker
Without further ado, let’s get started. You’ll need to install Docker and Compose first. (Selenium Grid is installed inside Docker, so don’t bother downloading it locally.) Luckily, both projects have great documentation on this topic:
https://docs.docker.com/installation/
https://docs.docker.com/compose/install/
Here’s an overview of what we’re going to do:
- Create Docker images for your Selenium Grid hub and node(s)
- Add Java to the hub to run the Selenium server jar
- Add Java, plus Firefox and Xvfb to the node
- Create a docker-compose.yml file to define how the images will interact with each other
- Start docker-compose and scale out to as many nodes as you need — or that your machine can handle
Create a Base Image for your Selenium Server and the Hub
Let’s start with creating the Dockerfiles for our Selenium Grid.
First off, both the require Java and the Selenium server jar, so let’s make a base Dockerfile we can use for both:
FROM ubuntu ENV VERSION 2.44.0 RUN apt-get update -qqy \ && apt-get -qqy --no-install-recommends install \ software-properties-common \ && rm -rf /var/lib/apt/lists/* RUN add-apt-repository -y ppa:webupd8team/java RUN echo debconf shared/accepted-oracle-license-v1-1 select true | debconf-set-selections RUN echo debconf shared/accepted-oracle-license-v1-1 seen true | debconf-set-selections RUN apt-get update -qqy \ && apt-get -qqy --no-install-recommends install \ oracle-java7-installer \ && rm -rf /var/lib/apt/lists/* RUN wget http://selenium-release.storage.googleapis.com/${VERSION%.*}/selenium-server-standalone-${VERSION}.jar
Let’s build and tag it:
docker build -t selenium/base .
The Grid is basically a single hub connected to any number of different nodes. So, our hub image should look something like this:
FROM selenium/base EXPOSE 4444 ADD start_grid.sh /var/start_grid.sh RUN chmod 755 /var/start_grid.sh CMD ["/bin/bash", "/var/start_grid.sh"]
And in the same directory as the Dockerfile, your run.sh script:
java -jar selenium-server-standalone-${VERSION}.jar -role hub
Install a Specific Version of Firefox
Finally, we need a single Dockerfile for all of our nodes. In this example, every node will be a copy of this Dockerfile. It follows the same format as above — start from our selenium/base image and install necessary apps: Firefox, Xvfb, etc. However, there’s one catch here. we need to install a specific version of Firefox. Why? Because Selenium is often incompatible with the last 1 or 2 versions of Firefox. (See the Selenium change log or their support policy for the most recently supported Firefox versions.) For this reason, specifying a specific version of Firefox in our Dockerfile is a good rule of thumb:
FROM selenium/base ENV FIREFOX_MINOR 34.0.5 RUN apt-get update -qqy \ && apt-get -qqy --no-install-recommends install \ firefox \ xvfb \ bzip2 \ && rm -rf /var/lib/apt/lists/* RUN [ -e /usr/bin/firefox ] && rm /usr/bin/firefox ADD https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/${FIREFOX_MINOR}/linux-x86_64/en-US/firefox-${FIREFOX_MINOR}.tar.bz2 /tmp/ RUN apt-get install -q -y libdbus-glib-1-2 RUN tar -xvjf /tmp/firefox-${FIREFOX_MINOR}.tar.bz2 -C /opt/ RUN chmod -R +x /opt/firefox/ RUN ln -s /opt/firefox/firefox /usr/bin/firefox ADD register-node.sh /var/register-node.sh RUN chmod 755 /var/register-node.sh CMD ["/bin/bash", "/var/register-node.sh"]
And our register-node.sh script:
#!/bin/bash xvfb-run --server-args=":99.0 -screen 0 2000x2000x16 -ac" \ java -jar selenium-server-standalone-${VERSION}.jar \ -role node \ -hub http://$HUB_1_PORT_4444_TCP_ADDR:4444/grid/register \ -browser browserName=firefox
What’s going on here? Well, we need both the Selenium server JAR as well as Firefox. And in order to have a display on which Firefox can run, we’ll need a virtual framebuffer, which Xvfb provides.
Define Your Selenium Grid Network with Docker Compose
Once we have these files created, let’s make a docker-compose.yml file in the same directory as our selenium_grid folder. The contents of the file should be as follows:
hub: image: selenium_grid/hub ports: - "4444:4444" firefox: image: selenium_grid/firefox links: - hub expose: - "5555"
Docker-Compose Up
From here, all we have to do is start it up:
#!/bin/bash docker-compose up -d docker-compose scale firefox=5
You should now have a Selenium Grid consisting of one hub and five Firefox nodes. (You can see them running here: http://<boot2docker_ip>:4444/grid/console)
To find boot2docker’s IP address, just run boot2docker ip. Once you’re there, you should see something like this:
Scale Up or Down
Want to scale your grid up or down? It’s easy: just type docker-compose scale firefox=20. Too much? Scale it back down to Goldilocks size with docker-compose scale firefox=10.
Need to nuke things from orbit and restart? Let’s stop everything. And for good measure, let’s completely remove the images too:
docker-compose stop docker-compose rm
From here, there are a lot of possibilities for setting up your Grid and integrating it with other Continuous Integration systems.
Questions? Comments? Leave me a comment below!
References
selenium/base
https://gist.github.com/jeato/c006e1306e4ba8a01cd5
selenium/hub
https://gist.github.com/jeato/59260b27ac60c3930c2c
selenium/firefox
https://gist.github.com/jeato/446240f00462956f81ec
docker-compose.yml
https://gist.github.com/jeato/b4d4e5a0248406b20485