Selenium: Exploring the Moon

Four months have passed since our first article about Moon — a Selenium-compatible browser automation solution created to work in Kubernetes or Openshift cluster. In this article I mostly described our motivation to build such solution and how to quickly try it. Today I would like to dive into the implementation details and explain how Moon works under the hood and why it is so efficient.

Moon Architecture

Moon is a first-class Kubernetes citizen and is always working inside it. Like any other application Moon is deployed as one or more pods and a service delivering network connectivity. Every pod includes:

  1. Moon — the main application providing Selenium API and by default listening on standard Selenium port 4444. Your tests should be launched against this port.
  2. Moon API — a supplementary component providing /status HTTP API and by default available on port 8080. Moon API returns additional cluster status information such as total cluster capacity, list of running browser sessions and so on.
  3. Moon UI — an optional graphical user interface visualizing cluster state retrieved from Moon API /status API. Currently we are using Selenoid UI as Moon user interface but more advanced solution is going to be delivered soon.

All these components are web-applications and in truly fault-tolerant cluster should have 2 or more replicas behind load-balancer. You can organize load-balancing using two Kubernetes primitives:

  • LoadBalancer — a generic L4 network load-balancer distributing IP-traffic across configured nodes.
  • Ingress — an L7 HTTP load-balancer with an ability to proxy requests to concrete HTTP URLs to specified services. For example you can give access to Selenium API in Moon service at /, whereas /status will lead to Moon API.

Internally Moon is using Kubernetes API to launch pods (and corresponding services) with browsers and then retrieve the entire cluster state. If you have already worked with Kubernetes — you should know that every service has a name which is also a network host name available in Kubernetes DNS. For browser pods Moon uses generated unique names containing information about launched browser, e.g. firefox-61-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840 which corresponds to Firefox 61.0. When Moon user requests a new browser session it receives created service name as a Selenium session ID instead of random UUID. This session ID is then sent in subsequent Selenium session requests and Moon is able to proxy the traffic to correct Kubernetes service with browser. Such simple idea makes Moon completely stateless and allows to start an unlimited number of replicas behind chosen load balancer.

What’s Inside Browser Pod

Moon is by default using just the same open-source browser images as Selenoid but you certainly can build and use your own. Let’s now take a look at typical browser pod and see what is inside it:

aandryashin:/$ kubectl describe po \
chrome-68-0-a607028a-3f25-4752-b463-94438cdd8aaf -n moon
Name: chrome-68-0-a607028a-3f25-4752-b463-94438cdd8aaf
Namespace: moon
Node: <none>
Labels: app=chrome-68-0-a607028a-3f25-4752-b463-94438cdd8aaf
Annotations: <none>
Status: Running
IP:
Containers:
video-recorder:
Image: aerokube/moon-video-recorder:1.2.0
# ...
logger:
Image: aerokube/logger:1.2.0
# ...
browser:
Image: selenoid/vnc_chrome:68.0
Ports: 4444/TCP, 6099/TCP
# ...
defender:
Image: aerokube/defender:latest
Port: 4545/TCP
# ...

Output above is a list of containers running in every browser pod when video recording and S3 upload are enabled. As you can see it consists of 4 containers:

  1. browser — an image with browser that actually executes Selenium tests. Is is exposing standard Selenium port 4444 and its X-server port 6099 for video recording purposes.
  2. defender — an utility proxying container guaranteeing that only one Selenium session is created in the pod. This container actively prevents (or defends) browser container from accepting the second new session request and also handles session timeout cases when pod is automatically deleted. Because of defender Moon does not need to store timeout state for every running session and thus remains completely stateless.
  3. video-recorder — a supplementary container with FFMpeg video processing software. It is used to capture the video from X-server when browser is actually running. When S3 uploading is enabled — this container additionally uploads recorded video file to S3.
  4. logger — a supplementary container used to upload browser session logs to S3 when it finishes.

So depending on Moon configuration and capabilities set in test code browser pod contains from two (browser and defender) to four containers.

Controlling Resources Consumption

The next important question I would like to talk about is browser resources consumption. Moon allows you to easily adjust guaranteed and maximum CPU and memory quantities for every running browser pod. This can be done globally for every browser version by providing a flag to Moon container: -cpu-limit, -cpu-request, -memory-limit and -memory-request flags. But is more flexible to control resources consumption for every browser version independently. For example Firefox is known to consume to a bit more CPU but Chrome needs more memory. Also resources consumption depends on the web application you are testing - sophisticated applications with tons of Javascript code definitely require more memory and CPU. We usually recommend to start with 1 CPU and 1 Gb of memory for every browser version and then decrease the value until you note performance degradation or page rendering issues. To set memory and CPU limits for browser version - update browsers.json:

{
"firefox": {
"default": "62.0",
"versions": {
"62.0":
"image": "selenoid/firefox:62.0",
"port": "4444",
"path": "/wd/hub",
"resources": {
"limits": {
"cpu": "2",
"memory": "2Gi"
},
"requests": {
"cpu": "200m",
"memory": "1Gi"
}
}
}
}
}
}

Running Moon in Isolated Environment

A frequent situation in big companies is Kubernetes running in isolated environment without access to Internet. This is mostly being done because of security reasons and Moon fully supports such mode of operation. To work in isolated environment you have to use your own internal Docker images registry to store Moon images. Having a working registry and network access to it from Kubernetes you have to reconfigure Moon to use this internal registry. This is done by specifying desired Docker images in browsers.json file to override browser image and in service.json file to override system containers such as logger or defender. For example to use custom browser images your browsers.json file can look like the following:

{
"firefox": {
"default": "62.0",
"versions": {
"62.0": {
"image": "my-registry.example.com/moon/firefox:62.0",
"port": "4444",
"path": "/wd/hub"
}
}
}
}

Overriding system images is done as shown below:

{
"images": {
"videoRecorder": {
"image": "my-registry.example.com/moon/video-recorder:latest-release"
},
"defender": {
"image": "my-registry.example.com/moon/defender:latest-release"
},
"logger": {
"image": "my-registry.example.com/moon/logger:latest-release"
}
}
}

These configuration files are then mounted to Moon pod as config maps and Moon is able to read and apply them.

Running Android Emulators in Kubernetes

The last question I would like to discuss in this article is how to run tests on “difficult” platforms such as Android. If you take a look at my Android-related article — you will understand that the most efficient and the least expensive way to run tests on Android platform is using Android emulators. Fast Android emulators compared to desktop browsers have an important particularity. Using virtualization technologies they require a certain number of processor instructions to be supported on the host where they are started. Such virtualization support usually exists either or hardware servers or virtual machines with nested virtualization support. And such machines are usually slightly more expensive than standard ones.

Fortunately Kubernetes can easily run on any heterogeneous set of hardware and virtual machines. The only thing we need to stay efficient is to be able to run Android pods on expensive hardware and the rest of browsers — on the cheap VMs. Such feature already exists and is called — node selectors. Every machine hosting Kubernetes is called a node and can be marked with a set of custom labels. When launching a pod you can set a requirement to start only on a node having some labels set. For example for Android you can mark every hardware node with expensive-hardware label and then configure Moon browsers.json file to follow such limitation:

{
"android": {
"default": "8.1",
"versions": {
"8.1":
"image": "selenoid/android:8.1",
"port": "4444",
"path": "/wd/hub",
"nodeSelector": {
"node-type": "expensive-hardware"
},
}
}
}
}

Now Android emulators will be running on such expensive hardware and the rest of browsers will be cheaper for you.

Still Having Questions?

We have several free support channels:

  1. Moon documentation: https://aerokube.com/moon/latest
  2. Our support email: support@aerokube.com
  3. Our Telegram support channel: https://t.me/aerokube_moon
  4. Our Github issues page: https://github.com/aerokube/moon/issues

Bonus: Free License Keys

We want to deliver a first-class polished Selenium solution and thus need more feedback from you. So here is how to get a free license key for desired number of parallel sessions:

  1. Open https://moon.aerokube.com/ and find Moon Evaluation License form.
  2. Fill your company name, an email address to send the license key and desired number of parallel sessions.
  3. Click on “Send me a key”.

You will then immediately receive a free license key valid until March, 1 2019 to the email specified.

Any More Articles?

Definitely. Take a look at these ones: