The Testing Library meets Selenium

The Testing Library enables testing as a user. It’s available for multiple JavaScript frameworks (e.g. React, Vue, Cypress). I realized it was possible and valuable to bring it to Kotlin’s Selenium.

The more your tests resemble the way your software is used, the more confidence they can give you. (Kent C. Dodds)

What

I developed a set of custom Selenium locators (e.g. ByRole, ByText) that wrap the corresponding Testing Library queries (specifically, the DOM Testing Library):

// with the Selenium Testing Library (Kotlin)
driver.findElement(ByAltText("First name"))
driver.findElement(
   ByRole("heading", name = "/as a user/i".asJsRegex())
)

// with the Testing Library (JavaScript)
screen.queryByAltText('First name')
screen.queryByRole('heading', { name: /as a user/i })

I believe the best examples are given by the tests, so check them out. As a sneak peek, here are some more examples of the Testing Library’s Core API:

val firstName = driver.findElements(ByAltText("first name"))
val person = driver.findElements(ByDisplayValue("/john/i".asRegex()))
val active = driver.findElements(ByLabelText("active"))
val input = driver.findElements(ByPlaceholderText("first name", exact = false))
val title = driver.findElements(ByRole("heading", name = "/as a user/i".asRegex()))
val panel = driver.findElements(ByTestId("test-id"))
val block = driver.findElements(ByText("present", exact = false, selector = "span"))
val title1 = driver.findElements(ByTitle("title 1"))

I also implemented the Testing Library User Interactions API:

driver.user.click(driver.findElement(ByRole("navigation")))
driver.user.type(driver.findElement(ByRole("textbox")), "foobar")
driver.user.selectOptions(driver.findElement(ByRole("listbox")), "C")
driver.user.keyboard("[/ShiftLeft][/ShiftRight]{Tab}")
driver.user.tab()
driver.user.clear(driver.findElement(By.Id("description")))
driver.user.pointer("[MouseLeft]", "[MouseRight]")

jest-dom is also available:

// API similar to the original version:
expect(button.toHaveAccessibleDescription("Register"))
expect(checkboxMarketing).toBeChecked()
assertEquals(setOf("btn", "btn-danger", "extra"), deleteButton.classes)
expect(element).not.toBePartiallyChecked()

// utilities that can be used on their own:
val formValues = registrationForm.formValues
val userAgrees = checkboxMarketing.isChecked
val name = element.accessibleName
val displayedValue = element.displayValue

I’ve packed all this into the Selenium Testing Library (for Kotlin, but it also works in Java) and open-sourced it.

Why

There was more than one reason to create this library:

• Contribute to the open source world, hoping to have created something useful for others;
• Dive deeper into Selenium, the Testing Library, and webpack;
• Learn how to publish a library to Maven Central (and perhaps how to automate it with GitHub actions);
• Use Selenium the way I believe UI testing should be done — testing as a user.

Why do I believe that testing as a user is superior to relying on technical aspects (ids, classes, tag names, test ids, XPath/CSS selectors)? It’s related to the goals of automated testing:

• The better the tests resemble the ways the users interact, the more confidence I get from them (note that this doesn’t imply end-to-end testing; the Testing Library can be used to test simply React components). I feel safer and more guided when refactoring if I’m able to change the underlying implementation details. CSS and XPath selectors and the HTML structure itself are all volatile low-level details. Test ids are a workaround, but that means creating implementation code solely for the sake of testing (a testing antipattern).
• If you, as a developer, have trouble finding UI elements, how can the users? When testing as a user, you’re compelled and encouraged to follow good usability practices like writing semantic HTML (e.g. article, section, nav) and providing accessibility labels (e.g. title, alt, aria-label).
• I get much more documentation power from the tests when they describe real-life usage scenarios. If I read those tests a few months later, I can easily relate to their goals and I don’t get lost in technical details.

Why not just use the Testing Library (in JavaScript or TypeScript) in the first place? I did it in some past projects (and that’s how I knew the Testing Library), but not everyone is working in those languages/runtimes. Some real-world projects are already running and using Selenium for JVM, and they also deserve the benefits of the Testing Library. Also, I appreciate what a compiled language can bring to a test suite.

Without this library, you’d have to resort to ids, classes, and test ids. If you didn’t want to depend on those technicalities, you’d need lots of utilities. In some cases, it wouldn't even be possible. For example, a case-insensitive-contains search could not be achieved with XPath (at least with current browsers).

<input placeholder='Username' />

// 🟠 With pure Selenium
val result = driver.findElements(By.cssSelector("*"))
    .first() {
        it.getAttribute("placeholder")?
          .contains("USER", ignoreCase = true) == true
    }

// 🟢 With Selenium Testing Library
val result = driver.findElement(
   ByPlaceholderText("USER", exact = false)
)

Beware that you still need to wait for the elements if they’re not immediately available (as with native locators); you can resort to Selenium’s waiting capabilities. However, if you prefer, you can rely on the Testing Library’s waits with an API that I added as an alternative:

val firstName = driver.queryBy(
   PlaceholderText, "first name", mapOf("exact" to false)
)

How

How have I integrated the Testing Library in Selenium? By evaluating its JavaScript so it’s available to be used later (equivalent to pasting and running it in the JavaScript console). Then, I can invoke the Testing Library functions and map the results back to Java/Kotlin objects. Here’s the juicy part of the technique:

// 1️⃣ in JS, store a handle to TL's screen:
import {screen} from '@testing-library/dom'
window.screen = screen

# 2️⃣ generate the TL script to be injected:
webpack --mode production
cp dist/testing-library.js ../lib/src/main/resources

// 3️⃣ evaluate the TL script so it gets stored in screen:
driver.executeScript(
   {}.getResource("/testing-library.js").readText()
)

// 4️⃣ call a TL function from screen and rely on Selenium to map the results to WebElements
val results = driver.executeScript(
   "return screen.queryAllByRole(arguments[0])", 
   "listbox"
)

As you can see, all the heavy lifting is done by webpack, the Testing Library, and Selenium. But there’s a missing piece: how to make these queries transparent to Selenium? I’d want to use them such as the native ones (e.g. driver.findElement(By.Id("profile"))). Selenium is easily expandable with new locators: inherit from By and implement findElements. Here’s an example of a custom locator (not from my library):

data class ByAttribute(private val name: String, private val value: String) : By() {
    override fun findElements(context: SearchContext): List<WebElement> =
        context.findElements(
            cssSelector("[$name='${value.replace("'", "\\''")}']")
        )
}

I can use that custom locator like any native Selenium locator: driver.findElement(ByAttribute("href", "http://example.com")).

Get, Query, or Find? One or All?

The Testing Library Core API has three families of queries: get* , query* , find*. Which one to pick? get* fails when elements are not found and I wanted to leave that for Selenium. find* waits until the elements are available, and again, Selenium has wait mechanisms. That leaves query* as the more flexible option and the one I used.

Each family of functions has a singular version (e.g. query) and plural version (e.g. queryAll). I used the plural because to create Selenium locators, we only need to implement findElements (plural). Selenium provides findElement (singular) based on it.

To port or to adapt?

Injecting the Testing Library script and bridging it to Kotlin/Java is known as the adapter pattern. However, my initial approach was to rely on Selenium locators (e.g. xpath, cssSelector) to port the Testing Library behavior aiming to achieve feature parity. It turned out this was a limited approach because:

• Selenium locators can’t compete with Testing Library search functions due to WebdriverIO and browser limitations (e.g. a case-insensitive text search by XPath);
• I’d have to port all functionality, which is a lot of work;
• I’d likely introduce bugs;
• I wouldn’t get library updates easily.

I ended up using the injection approach, which is valid based on the MIT license of the Testing Library. An exception to this was that I ported jest-dom rather than bridging it. To bridge it, I had to inject jest and jest-dom which proved to be troublesome.

Won’t it be slow?

Injecting the Testing Library has a performance impact. However, consider the following optimizations:

• The source JS scripts are minified (each is around 250KB).
• The JS files are read at most once from the filesystem after which they’re read from memory.
• Each script is independently injected at most once (until there’s a page navigation or refresh) and only if/when needed.

The injection times roughly vary between 20 and 40 milliseconds. Considering the benefits of the Testing Library, I believe it pays off.

Here’s the library’s source code:

GitHub - lsoares/selenium-testing-library: Selenium locators for Java/Kotlin that resemble the…

Here’s the library published at Maven Central:

Maven Central Repository Search