UI Test automation on Web portals under Bot Detection

Photo by Shahadat Rahman, Unsplash

Nowadays test automation technologies have evolved to a significant degree. Especially in the area of web UI test automation, architects and automation engineers have a vast variety of frameworks to choose from (Selenium Webdriver, Cypress, etc). However, automated testing of a web portal which is equipped with anti-bot technology presents a significant challenge. Such a web portal will, by design, behave differently when accessed by an automated tool vs when accessed by a human.

But what exactly is an anti-bot?

Anti-Bot technology is a mitigation process put in place to stop bad bots. Bad or “malicious” bots are targeting businesses in all industries including financial services, retail, travel, and gaming using a variety of automated techniques. Amongst the most common automated bot attack strategies are credential stuffing, card cracking, web scraping, and ad fraud.

At this very point, the challenges for the test automation organization begin. Test automation is like another kind of bot. A bot that simulates the user in order to test if everything works as expected. How can we resolve this conflict between the legit automation tests trying to verify the correctness of the application and the bot-detection functionality that tries to protect itself from the bot? In this article, we will review some of the approaches and/or solutions we can adopt and follow in our testing strategy.

Bypass it

Starting with the obvious one: we could disable the bot-detection functionality just during internal testing. Although this option sounds very appealing it comes with a potential cost on quality.

There are numerous times, especially in the case where the anti-bot is deployed as a JavaScript library embedded in the web portal, where the anti-bot causes many user experience (UX) issues and/or even functional problems. If we disable the anti-bot, these issues will not be found during automation testing. So this strategy should not be adopted and if so always plan for an extra test cycle with the anti-bot enabled prior to release.

Act as human

Our automation tests are basically screaming to every page they visit: “I’m a bot!”. Selenium, for example, will take actions before HTML renders, take numerous actions in an instant — things that humans won’t do. For example, no human being can complete & submit a large form in less than a second.

So as a starting point we need to enforce our tests to act like humans. This can be easily accomplished by having a random delay from 3 to 8 seconds before each action takes place. For example, something like time.sleep(random.randint(3,8)) will be enough for the majority of the anti-bot platforms.

Moreover, we need to modify HTTP request headers such as the User-Agent value, which by default indicate that the actions are driven by a script and not by a human. Sample code for implementing such changes in a Chrome browser can be found in the following snippet:

driver = webdriver.Chrome()
driver.execute_cdp_cmd(‘Network.setUserAgentOverride’, {“userAgent”:”myAgent”, “platform”:”Windows”})
driver.get(‘http://mysite.org')

Hack it

In some rare cases Websites can detect automation scripts using JavaScript experimental technology, such as the navigator.webdriver property in the Navigator interface. When the website is accessed by automation tools (mostly Selenium), the value of navigator.webdriver is set to true.

To prevent Selenium driven WebDriver from getting detected a nice approach would include the following steps:

• Change the property value of the navigator for webdriver to undefined

driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

• Exclude the collection of enable-automation switches:

options.add_experimental_option("excludeSwitches", ["enable-automation"])

• Turn-off useAutomationExtension:

options.add_experimental_option('useAutomationExtension', False)

Alternatively, a quick & dirty solution is to tweak the webdriver executable.

Basically the way the Selenium detection works is that the anti-bot tests for pre-defined JavaScript variables that appear when running with selenium. The bot detection scripts usually look for anything containing the word “selenium” / “webdriver” in any of the variables (on window object), and also document variables called $cdc_ and $wdc_. Of course, all of this depends on which browser you are on. Each browser exposes different things.

For Chrome, all we have to do is to ensure that $cdc_ does not exist anymore as a document variable. To do so take chromedriver.exe into a hex editor and search for $cdc. We will find something like "$cdc_asdjflasutopfhvcZLmcfl_’. Replace the whole part after the $ with another string of exactly the same character length. Save and use this executable as a driver from now on.

These tips are a good starting point for making your automation testing work nicely with the bot-detection algorithms.

Happy testing!