Published in


The future of user testing with automation + AI

User testing has been around as long as software development has been around. For the past decade, the advancement of automation technology gave us the ability to perform user testing faster, more thoroughly and more efficiently. More recently, the explosion of machine learning/AI raises more interesting prospects on adding AI application on top of testing automation to further reduce the involvement of QA engineer in user testing process.

To illustrate this, let’s set our context in user testing for websites or web-based applications.

Example e-commerce website

For example, we might define some testing scenarios when performing user testing on this e-commerce website:

  • The user should be able to search for products
  • The user should be able to check out products
  • The user should be able to make payment with credit cards
  • Etc Etc.

Generally, a number of test cases will be defined based on business priority, time and resource constraints from the technical team.

Manual testing

Once the test cases (and criteria for passing tests) are defined, a manual tester can go through each testing step and make sure the test case/scenario work as expected. This generally happens when the website is updated or new software code is deployed.

  • Advantages of manual testing:
    low barrier to perform the test, humans can execute complex tests.
  • Disadvantages
    Potential human error
    Labor cost

Manual testing is still around today for tests that are difficult to automate as well as urgent/time sensitive tests. There are also a number of crowdsourced manual testing services available to offset some of the disadvantages.

Automated testing

Automated testing transforms defined test cases into a repeatable process that can be executed whenever there’s a new code deployment (update) to the website.

Think of automated testing as using an application such as the Automator on the Mac

Generally speaking, there’s a specific technology that executes the test cases and test engineers need to write tests (scripts) that correspond to the defined test cases. The tests usually consist of the steps in the test case as well as expected outcomes (assertions). For web-based user testing, one of the popular technology is “Selenium

  • Advantages of automated testing:
    Have broader test coverage (test more)
    Eliminate human error
    Less human resource needed
  • Disadvantages
    Maintenance of test scripts
    Testing automation technology skill requirement
    Technology change/adoption
    Complex tests

There are few additional hidden drawbacks that are not often thought of in automated testing:

  • Inability to test all variations of test cases.
  • Provides team a false sense of security because tests are “passing”. However, test automation only goes as far as the tests cases we define.
  • Lack of consideration for user experience. Automated user testing is binary in nature, they either “pass” or “fail”. They don’t care how a user feels when he/she goes through the process in a live environment.

Automated testing with AI

Photo by Josh Riemer on Unsplash

In large part, an AI-based automated user testing system can be developed through “bots” that can be trained to understand how to navigate websites and web-based applications.

We already employ “bots” to perform tasks such as crawling websites to index their contents, automate certain interactions. However, most bots that perform specific tasks such as screen scraping or form submission has to be explicitly programmed to perform tasks that are well defined.

A truly AI-based bot will be trained to understand elements on the web as well as how to navigate through them (actions). For example:

Elements such as

  • Links
  • Buttons
  • Form element

Actions such as

  • Page navigation
  • Form submission
  • Online shopping/checkout

Data source

The biggest challenge in any AI system is the amount of labeled data available for supervised learning. Luckily in our case, we have an abundance of data through analytics tools.

For example, google analytics provides detailed user behavior flow that can be used as labeled data for defining automated test cases and used as training data for performing different actions through a number of paths.

We can retrieve video recordings from services such as Hotjar and provide training labels for actions accomplished through a specific user session.


The biggest benefit of an AI-based automated user testing system is that we leverage real/live data from actual users as our test cases instead of defining them ourselves.

Additionally, the system can help generate additional test case variances that are inconceivable for QA engineers to come up with, this greatly increases our test coverage and robustness.

Finally, we no longer need to write as well as maintain test scripts. The primary tasks for QA engineers will be to create a robust data collection and labeling process for creating tests.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Chang Xiao

Chang Xiao


Starter, dev, digital consultant, cyclist, tennis player. Currently focused on data science and specifically recommendation systems.