EXPEDIA GROUP TECHNOLOGY — SOFTWARE

Getting started with a Dockerized Selenium Grid in your On-Prem Data Centers

This unopinionated article aims to help you start a UI functional testing platform in a cost-efficient yet scalable way

Photo by Marvin Meyer on Unsplash

Automated UI functional testing on a Selenium Grid is a useful tool for software engineers, but historically it has required a lot of complicated setups. Recently, Expedia Group™ has developed a testing platform (described more fully in Ambighananthan Ragavan’s post, DaKube) that makes use of containerization to make it easy to scale UI functional testing clusters automatically based on demand.

In this article, we show how to set up and use Docker with Selenium Grid container images to implement a simplified version of Expedia’s testing system. We’ve boiled that system down to the basic elements so that you can get familiar with those elements in an unopinionated setting — you can follow these steps on your own on-premises cluster or using cloud-based instances.

Prerequisites

1. Basic understanding of a Selenium Grid
2. Pre-installed Docker machines.
3. VNC Viewer or a similar tool
4. Basic understanding of docker commands (command references will be linked along with usages)

Introducing “The Containerized Grid”

Standing and maintaining a selenium grid is a considerable effort. Especially, when we work with hundreds of WebDriver nodes for the sake of concurrency. Thanks to containerization, Selenium now offers a robust, hassle-free solution. Introducing the components of a “Containerized Selenium Grid”:

• Selenium hub: https://hub.docker.com/r/selenium/hub
• Selenium Chrome Node: https://hub.docker.com/r/selenium/node-chrome
• Selenium Debug Chrome Node: https://hub.docker.com/r/selenium/node-chrome-debug
• Selenium Firefox Node: https://hub.docker.com/r/selenium/node-firefox
• Selenium Debug Firefox Node: https://hub.docker.com/r/selenium/node-firefox-debug

Debug Vs Non-Debug Nodes: As the name suggests, “debug” nodes are meant to be the development nodes that would produce verbose logs. You will also be able to view the running tests live on a debug node using a VNC tool.

You can stand the grid on one node or on multiple nodes based on your need. The following will be explained in detail in this article:

1) Standing the grid on a single instance/dev box.

2) Standing a distributed grid.

Standing the grid on a single instance/dev box

• Start a docker hub by executing $ docker run -d -p 4446:4444 --name selenium-hub selenium/hub. Note that 4446 is used as the port to stand the hub in the above command. You can view the hub’s console at <host>:<port>/grid/console. Docker Reference: docker run

Fig. 1 Selenium hub without nodes

• Attach a Chrome node and a Firefox node to the hub

$ docker run -d --link selenium-hub:hub -v /dev/shm:/dev/shm selenium/node-chrome
$ docker run -d --link selenium-hub:hub -v /dev/shm:/dev/shm selenium/node-firefox

Fig. 2 Selenium hub with a Chrome and a Firefox node

Now the hub is ready to take a Chrome test and a Firefox test in parallel*.

• You can also increase the number of browsers per container by setting NODE_MAX_INSTANCES

$ docker run -d --link selenium-hub:hub -e NODE_MAX_INSTANCES=5 -v /dev/shm:/dev/shm selenium/node-chrome
$ docker run -d --link selenium-hub:hub -e NODE_MAX_INSTANCES=5 -v /dev/shm:/dev/shm selenium/node-firefox

Fig. 3 Selenium hub with multiple browsers

If the hardware is sufficient*, you already have a hub that is capable of running 5 Firefox tests and 5 Chrome tests in parallel.

*Hardware could prove to be a bottleneck for test concurrency. Refer to the “Concurrency Bottleneck” section at the end of the article.

Standing a distributed grid

When we were running the hub and nodes on the same machine, we were able to link the containers using the--link parameter which is now deprecated by docker. This will be good enough for POCs but not for the production load that has to scale and probably run millions of tests every day. Let’s take a look at standing a distributed grid.

What is a distributed grid?

A distributed selenium grid is created by standing the hub and nodes within a cluster of machines that can communicate with one another through IPs or domain names.

Fig. 4 Selenium Grid publisher-subscriber connection. Disclaimer: The function names shown above are just for understanding purposes and are not true.

In such a distributed grid, a selenium node has to perform the following actions.

1. Correctly identify the Selenium hub
2. Register itself with the hub
3. Provide the hub details on how to communicate back to the connecting node to establish successful communications. Note: Since the nodes are containerized, the IP that the node shares with the hub by default will not be accessible from anywhere outside the instance where the node is running.

A dig at the environment variables

A containerized node image will use the following key environment variables. Understanding them is very important to correctly set the grid up and maintain it over long periods.

HUB_HOST: Indicates the host name of the HUB
HUB_PORT: Indicates the port number of the HUB
NODE_HOST: Indicates the host name of the node that the hub could use to communicate back
NODE_PORT: Indicates the port number of the node that the hub could use to communicate back
REMOTE_HOST: Indicates http://<NODE_HOST>:<NODE_PORT> which replaces NODE_HOST and NODE_PORT

Distributed setup example

Let’s try to understand this through an imaginary example…

Note: The below example can be performed on any on-prem machines and do not have to be on EC2. I have chosen EC2 for this example.

Bob, a Platform Engineer at Imagination Inc., decides to create a distributed selenium grid for a POC. He wishes to try and stand a grid similar to Fig.3 but in a distributed setup (i.e the hub, the Chrome node, and the Firefox node will all have to run on different machines).

To achieve the distributed cluster, he creates three EC2 instances with the following specs. Specs: t2-micro, Ubuntu 18.04 LTS, Docker installed, security group configured to allow incoming and outgoing HTTP connections on ports 4444 and 5555 (How to create these EC2 instances is not within the scope of this article). He then decides to set up his hub and adds the following note for his reference.

######## Bob’s Note Starts…. ########

IP addresses of my cluster (all the below IPs are imaginary),

1. 18.236.100.1 (hub)
2. 18.236.100.2 (chrome-node-1)
3. 18.236.100.3 (firefox-node-2)

Starting the hub: SSH into 18.236.107.1 and run the following command

ubuntu@ip-18-236-100-1 $ docker run -d -p 4444:4444 selenium/hub:latest

This will start a hub on the EC2 instance (IP = 18.236.100.1 and Port = 4444).

Chrome-node-1: SSH into 18.236.100.2 and run the following command

ubuntu@ip-18-236-100-2 $ docker run -d -p 5555:5555 -p 5900:5900 -e NODE_MAX_INSTANCES=5 -e REMOTE_HOST="http://18.236.100.2:5555" -e HUB_HOST=18.236.100.1 -e HUB_PORT=4444 selenium/node-chrome:latest

This will start a Chrome node on the EC2 instance (IP = 18.236.100.2 and Port = 5555).

Firefox-node-2: SSH into 18.236.100.3 and run the following command

ubuntu@ip-18-236-100-3 $ docker run -d -p 5555:5555 -p 5900:5900 -e NODE_MAX_INSTANCES=5 -e REMOTE_HOST="http://18.236.100.3:5555" -e HUB_HOST=18.236.100.1 -e HUB_PORT=4444 selenium/node-chrome:latest

This will start a Chrome node on the EC2 instance (IP = 18.236.100.3 and Port = 5555).

Note: $REMOTE_HOST is the environment variable that the node will use to register itself to the hub. If this URL does not correctly self reference the connecting node, the hub will never be able to reach the node back and the connection will not succeed.

Fig. 5 Distributed Grid

######## End of Bob’s Note! ########

Debugging

To debug and view a running test, we should use “debug” docker nodes.

Let’s add a debug Chrome node to the hub from one of the VMs.

$ docker run -d -p 5555:5555 -p 5900:5900 -e NODE_MAX_INSTANCES=5 -e REMOTE_HOST=”http://<node-ip>:<node-port>" -e HUB_HOST=<hub-ip> -e HUB_PORT=<hub-port> selenium/node-chrome-debug:latest

Install VNC Viewer or some other VNC tool and open the same

Fig. 7 VNC Viewer Console

From VNC Viewer, connect to your debug node by <ip>:5900.

Note: VNC Viewer will ask you to key in a password. It will use “secret” as your default password till you change it.

You will then see a console like Fig.7 where the executing selenium tests could be seen.

Note: VNC port by default will be 5900.

A Smoke Test

A simple test like this could be written to smoke test the platform. Note: The following test is written using Node.js using npm. The same could be written with other supported languages too.

• Create a folder like the following $ mkdir find-expedia-test
• Add the following two files inside the folder

Fig. 6 find-expedia-test

• Run the following $ npm install and notice that a node_modules folder gets created after npm install is complete.
• Now, run the test by $ node find-expedia-test.js

When we connect to the VNC viewer on the Chrome node debug instance, we will see a running test like the following

Fig. 8 A running test viewed from VNC’s console

Concurrency Bottleneck

Please note that provisioning x nodes do not mean that the execution of the tests will be concurrent by a factor of x. Overprovisioning of nodes will lead to excessive context switching of processes and add unnecessary overhead.

The following formula could be used to find out the number of concurrent browsers to spin on each node for maximum scalability.

100 / CPU Utilization per core * Number of available cores

Advantages

• Extremely Scalable. This solution could easily be optimized to run millions of tests every day.
• Reusable and cost-efficient. With proper design and implementation, the same set of machines could host Chrome browsers and later switch to Firefox browsers through docker API saving the need to have dedicated hosts for browsers.

Limitations

• Unfortunately, we cannot create a hub with IE or Safari browsers due to the non-existence of docker images for these browsers.
• Maintenance of the grid, especially if we choose to customize the images.
• Images should be security scanned since they are maintained by the Selenium Community.

Conclusion

The steps mentioned in this article are unopinionated and will be particularly useful when you cannot use any other container orchestration frameworks or cloud service providers. You can follow this article to create a selenium grid on any cluster of VMs including your on-prem machines.

If you choose to build your own docker based selenium grid as the article suggests, I would surely recommend carefully weighing the advantages and limitations that are outlined above.