K8s Selenium Grid — Selenium Grid with Docker on Kubernetes

5 min readNov 26, 2018

Getting Started

Kubernetes is a platform for hosting Docker containers in a clustered environment with multiple Docker hosts.
We are gonna be running a single node Kubernetes cluster using Minikube

Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to give it a try Kubernetes use cases as POC’s

Prerequisites

First we need to setup Kubernates cluster locally on your computer, below is the link to setup using minikube

Setting up Kubernates cluster locally using MiniKube

There are several ways of orchestrating a selenium grid using kubernetes by choosing your own cloud solutions

Here I’m going to explain about minikube and google cloud

1. K8S selenium grid setup using Minikube

Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users who are looking to give it a try some kubernates business use cases for POC’s

Launching Minikube after single node K8s cluster setup

Minikube running status

Available K8s nodes

Selenium Docker latest images could be found here in this below link

Docker images for Selenium

Launching selenium hub as a service on k8s cluster

$ kubectl run selenium-hub --image selenium/hub:3.10.0 --port 4444
deployment "selenium-hub" created$ kubectl get pods
NAME                          READY     STATUS    RESTARTS   AGE
selenium-hub-6bdc9c44df-715t8   1/1       Running   0          43s

Spin up Chrome node as deployment on k8s cluster

$ kubectl run selenium-node-chrome --image selenium/node-chrome:3.10.0 --env="HUB_PORT_4444_TCP_ADDR=selenium-hub" --env="HUB_PORT_4444_TCP_PORT=4444"
deployment "selenium-node-chrome" created$ kubectl get pods
NAME                    READY     STATUS              RESTARTS   AGEselenium-hub-715t8       1/1       Running             0          7mselenium-node-chrome-lqkh2 0/1       Running             0       31s

Spin up Firefox node as deployment on k8s cluster

$ kubectl run selenium-node-firefox --image selenium/node-firefox:3.10.0 --env="HUB_PORT_4444_TCP_ADDR=selenium-hub" --env="HUB_PORT_4444_TCP_PORT=4444"deployment "selenium-node-firefox" created$  kubectl get pods
NAME                    READY     STATUS              RESTARTS   AGEselenium-hub-6bdc9c44df-715t8      1/1       Running             0          7mselenium-node-firefox-6fb58c9fcf-lqkh2 0/1       Running             0       31s

Scaling up deployments

Scale up selenium chrome nodes

$ kubectl scale deployment selenium-node-chrome --replicas=4
deployment "selenium-node-chrome" scaled$  kubectl get pods
NAME                                    READY     STATUS   RESTARTS      AGEselenium-hub-6bdc9c44df-715t8           1/1       Running        0          7mselenium-node-chrome-6fb58c9fcf-lqkh2   1/1       Running        0          7mselenium-node-chrome-6fb58c9fcf-k7lz9   1/1       Running        0          10sselenium-node-chrome-6fb58c9fcf-8sdz7   1/1       Running        0          10sselenium-node-chrome-6fb58c9fcf-s27tt   1/1       Running        0          12s

Scale up selenium firefox nodes

$ kubectl scale deployment selenium-node-firefox --replicas=4
deployment "selenium-node-firefox" scaled
$  kubectl get pods
NAME                                    READY     STATUS    RESTARTS   AGEselenium-hub-6bdc9c44df-715t8           1/1       Running        0          7mselenium-node-firefox-cf9c5b7bc-5t5zz   1/1       Running        0          7mselenium-node-firefox-cf9c5b7bc-fgtc7   1/1       Running        0          10sselenium-node-firefox-cf9c5b7bc-sx9cq   1/1       Running        0          10sselenium-node-firefox-cf9c5b7bc-xsqvr   1/1       Running        0          12s

Expose selenium hub to be accessible

$ kubectl expose deployment selenium-hub --type=NodePort
service "selenium-hub" exposed$ kubectl get services
NAME           CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
kubernetes     10.96.0.1     <none>        443/TCP          2m
selenium-hub   10.107.134.138   <nodes>    4444:32629/TCP   46s

Selenium hub url can be accessed through

$ minikube service selenium-hub --url
http://192.168.99.100:31138/

Chrome and Firefox instances after scale up from Selenium grid console

Minikube Dashboard

$ minikube dashboard

2. K8S selenium grid setup using Google Container Registry

Yaml image files for Selenium

Google Container Engine is also a quick way to get Kubernetes up and running: https://cloud.google.com/container-engine/ Your cluster must have 4 CPU and 6 GB of RAM to complete the example up to the (4 instances) scaling portion.

Launching selenium hub as a service

$ kubectl create -f selenium-hub-deployment.yaml

Lets create a service for nodes to connect to,

$ kubectl create -f selenium-hub-svc.yaml

Spin up Chrome & Firefox Nodes

Now since hub is up and exposed lets spin up nodes and register into hub

$ kubectl create -f selenium-node-chrome-deployment.yaml
$ kubectl create -f selenium-node-firefox-deployment.yaml

Now nodes are up and registered to hub created above can be seen in exposed selenium-hub’s console.

Self-Healing Capability

One of the greatest adavantage of kubernetes is Self healing.Though docker-swarm also supports self-healing, kubernetes is more reliable solution over a period of time due to replication controllers. To demonstrate lets get the list of pods created,

$ kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
selenium-hub-6m8v1           1/1       Running   1          2d
selenium-node-chrome-k146x   1/1       Running   0          2d
selenium-node-chrome-m4zt6   1/1       Running   0          2d

Let’s delete a google chrome node using below command,

$ kubectl delete pod selenium-node-chrome-m4zt6
pod "selenium-node-chrome-m4zt6" deleted

Now again get list of pods (chrome nodes)

$ kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
selenium-hub-6m8v1           1/1       Running   2          2d
selenium-node-chrome-k146x   1/1       Running   0          2d
selenium-node-chrome-sx9cq   1/1       Running   0          41s

To look at the logs on specific pod

$ kubectl logs selenium-node-chrome-k146x
12:00:25.706 INFO [GridLauncherV3.launch] - Selenium build info: version: '3.10.0', revision: '176b4a9'12:00:25.818 INFO [GridLauncherV3$3.launch] - Launching a Selenium Grid node on port 55552018-11-20 12:00:26.701:INFO::main: Logging initialized @4212ms to org.seleniumhq.jetty9.util.log.StdErrLog12:00:28.304 INFO [SeleniumServer.boot] - Selenium Server is up and running on port 555512:00:28.304 INFO [GridLauncherV3$3.launch] - Selenium Grid node is up and ready to register to the hub12:00:28.310 INFO [SelfRegisteringRemote$1.run] - Starting auto registration thread. Will try to register every 5000 ms.12:00:28.310 INFO [SelfRegisteringRemote.registerToHub] - Registering the node to the hub: http://selenium-hub:4444/grid/register12:02:01.865 INFO [SelfRegisteringRemote.registerToHub] - Registering the node to the hub: http://selenium-hub:4444/grid/register12:02:02.132 INFO [SelfRegisteringRemote.registerToHub] - Updating the node configuration from the hub12:02:02.230 INFO [SelfRegisteringRemote.registerToHub] - The node is registered to the hub and ready to use

We can see that within few seconds selenium hub replication controller automatically generates a new chrome node or pod selenium-node-chrome-sx9cq and assigns itself. Replication Controller is one of the greatest asset of kubernetes to manage nodes automatically in a cluster.

Sizing and Memory management in K8s cluster

Ideal memory for Kubernates pods must be like 1.5 GB for Chrome and Firefox

“resources”: {
“limits”: {
“cpu”: “500m”,
“memory”: “1536Mi”
}
}

Advantages:

quick one time setup and more scalable solution
less maintenance and high availability
self healing capability
easy to update for versioning changes
eradicates latency issues as it resides in same network