Selenium Grid 4 setup @OneShop
After setting up a flourishing automation ecosystem for OneShop on Stark, it was time to invest more into test execution infrastructure because, at Deutsche Telekom Digital Labs, we always aim to build & test products that have a resilient future.
As Tim Berner-Lee says
“The Web as I envisaged it, we have not seen it yet. The future is still so much bigger than the past.”
Well, Tim is right.
Why settle with the existing tech stack?
Why not choose what’s growing?
In fact, why not contribute back to the testing community with ease of documentation and sharing the lessons learned.
We at OneShop are executing multiple automation nightly jobs for our multitenant OneShop. One framework, one automation script, multiple webshops tested at one go.
Read more in the above article, how are we going about it.
In addition, now, the framework supports both actual device and mobile emulation.
With that said, moving from Selenium 3 to 4 had two primary phases.
PHASE I
In mid-2021, our journey of moving towards Selenium 4 started, and code fully migrated in a couple of months.
PHASE 2
Phase 2 meant shifting executions from
In-house test servers(Mac Minis) to
AWS cloud-based VMs running ̶S̶e̶l̶e̶n̶i̶u̶m̶ ̶G̶r̶i̶d̶ ̶3̶ Selenium Grid 4.
After reading numerous articles and watching countless repeated tutorials to understand the latest of Selenium Grid, it took some time to stabilize our deployment of Grid, which is now just a Jenkins job away.
We wanted to be future-ready, and indeed Selenium Grid 4 comes with a fresh face & set of new features.
The Mighty Setup
The below-mentioned information is not easy to find unless you decide to spend 60–70 hrs of research to connect some missing pieces present in the official documentation here.
YES! It means we should start contributing to the above link, or you can start by liking & commenting on this article below to spread love & learning.
With that said,
“Yeah, here we go for the hundredth time
Hand grenade pins in every line
Throw ’em up and let something shine!”
Step 1: Getting Started
Please read the official documentation about Selenium Grid 4 and what is new in it. How does it differ from the old version?
Step 2: Pre-requisite checks
Now that you know what Selenium Grid 4 is, it’s time to do a hands-on setup. I will be sharing steps directly about the distributed design and not about the standalone or hub way because that is pretty straightforward.
Download. Run. Observe.
https://www.selenium.dev/documentation/grid/getting_started/
Before proceeding to the below-mentioned steps, in this step, you need to evaluate
a. Whether you even need a distributed setup?
b. Do you even have sufficient resources(pre-requisite) to run a distributed setup (2 VMs on any of the cloud(AWS/GoogleCloud/DigitalOcean))
If the answer to both above is a YES, go ahead.
Step 3: Downloading necessary files
Download the latest JAR file from here and upload them to both the Virtual Machines you will use to set up a distributed selenium grid.
Alternatively, you can also run to download via command line
wget https://github.com/SeleniumHQ/selenium/releases/download/selenium-4.0.0/selenium-server-4.1.2.jarmv selenium-server-4.1.2.jar se4.jar // Renaming for ease of run
Step 4: Managing & storing JARs
I would recommend moving this download jar to somewhere accessible by multiple users as a matter of practice. e.g. /opt directory
cd /opt
mkdir /selWeb
mv /Downloads/selenium-server-4.1.2.jar .Do this in both the machines.
Step 5: Naming the VMs
For the sake of understanding, let’s name the two VMs as
Machine1: DisMaster
Machine2: DisNode
As per the terminology, we will be setting up all queues and the router on DisMaster, whereas DisNode will act as the Node responsible for running actual tests.
Step 6: Port Awareness & Relaxation
In distributed Selenium Grid Setup, we need to set up two-way communication between DisMaster and DisNode on multiple ports. This calls out for the relaxation of specific ports for incoming & outgoing traffic.
For the starters,
a. Enable/allow incoming connections from DisNode on port numbers 4442, 4443 on the DisMaster
b. Enable/allow incoming connections for the port ranging from 5550 to 5559 on both DisMaster, DisNode.
c. Enable incoming connections on the grid URL node of your choice. It can be any port, and the default is 4444. We will use 5550 as a port because 4444 was running an old instance of Selenium Grid 2.
Step 7: Selenium Grid Components Glimpse
This is important to understand the flow of data and understand the underlying components.
Must read: https://www.selenium.dev/documentation/grid/components/
Step 8: Start the Event Bus
Assuming you are in the same directory where the selenium jar is present. Create a TMUX session and run
tmux new -s ebjava -jar se4.jar event-bus
It serves as a communication path to other Grid components in subsequent steps.
It runs on port 5557 by default.
Note: I will be using tmux sessions before every command so that we can maintain sessions in parallel and observe what’s happening whenever needed.
Step 9: Session Map
tmux new -s sessionsjava -jar se4.jar sessions
A session map is responsible for mapping session IDs to the Node where the session is running.
It runs on port 5556 by default.
Step 10: Let’s form a queue of sessions!
tmux new -s sqjava -jar se4.jar sessionqueue
Session Queue adds the new session request to a queue; then the distributor processes it.
It runs on port 5559 by default.
Step 11: The DISTRIBUTOR
tmux new -s distributorjava -jar se4.jar distributor --sessions http://localhost:5556 --sessionqueue http://localhost:5559 --bind-bus false
Nodes register to it and assign a Node for a session request.
It runs on port 5553 by default.
Any mishandling of the same can lead to…
Step 12: The Mighty Routing Begins
It’s now time to run the router, which is the grid entry point, the address used in your automation code.
The router is in charge of routing requests to the correct component.
It runs on port 4444 by default.
A different one can be specified by -p parameter, 5550 in our case.
tmux new -s routerHubjava -jar se4.jar router --sessions http://localhost:5556 --distributor http://localhost:5553 --sessionqueue http://localhost:5559 -p 5550
Once the router has started, you will see such a screen.
So the Grid has started with the final step of the router, but there are no nodes yet. If you try to run a test, you will encounter SessionNotStarted exceptions & similar.
Step 13: Connecting a Node
Well, as scary as the number 13 sounds, the same horror was there to properly master this step of connecting a node properly to the DisMaster.
DisNode → DisMaster
So login/ssh into the DisNode machine.
Pre-requisite steps: Step 3 and Step 4 above.
tmux new -s nodejava -Dwebdriver.chrome.driver=/usr/local/bin/chromedriver -jar se4.jar node --publish-events tcp://{DisMasterPublicIPAddress}:4442 --subscribe-events tcp://{DisMasterPublicIPAddress}:4443 --grid-url http://{DisMasterPublicIPAddress}:5550
{DisMasterPublicIPAddress} would be the public IP address of the VM running the router.
The Node needs to connect to the router machine and subscribe to publish and subscribe events running on ports 4442 and 4443. If you don’t perform this mapping, NOTHING will work. Trust me!
Step 14: Selenium 4 Grid in action
Visit http://{DisMasterPublicIPAddress}:5550/ui/index.html#/ in your favourite browser and witness the Node, the router, and the GRID4 in action.
Well, this is it. This is your moment of glory.
. >
. >>
. >>>
. >>>>
.>>>
.>>
.>
Step 15: And you thought it was done.
Update your code with the new URL, and with Grid4, you don’t need to mention /wd/hub in the remote driver URL
String oldCodeOldHub = "http://localhost:4444/wd/hub";// Notice there is no need of /wd/hub
String newCodeNewLife = "{DisMasterPublicIPAddress}:5550";public WebDriver driver;
driver = new RemoteWebDriver(new URL(newCodeNewLife), caps);
Step 16: Sing along like Adele would do.
“This is the end
Hold your breath and count to ten
Feel the Earth move and then
Hear my heart burst again
For this is the end
I’ve drowned and dreamt this moment
So overdue, I owe them
Swept away, I’m stolen.
Let the tests RUN,
When it crumbles
We will stand tall
Face it all together, let the tests run”
Up next: We plan to use Grid 4 in a dockerized way with custom dashboards, reducing the setup effort & maintenance, stay tuned!
