Resolving 6 key Jetpack Compose UI testing problems

Suggestions for Enhancing the Google Testing Framework

For years, the Android development used frameworks like UI Automator and Espresso to test applications built around the View component. Despite their utility, these frameworks had several issues, leading to the emergence of open-source wrappers to address them.

The Android world underwent a revolution with the introduction of Jetpack Compose. Alongside it, Google introduces Compose UI testing framework (actually, there is no official name for Compose testing framework from Google but lets use this one). During the transition from Espresso to the Compose UI testing framework, automation testing methodologies across platforms had advanced significantly. Frameworks like Playwright in the web domain showcased the evolution of native test development, incorporating best practices and addressing common automation challenges.

With the release of the Compose UI testing framework, there were optimistic expectations that Google would take into account the community’s experience and integrate best practices into the new testing framework. Unfortunately, many issues remained unresolved, and some even emerged.

In this article, we explore six key problems encountered with the Compose UI testing framework and propose solutions drawn from another open-source framework Ultron. While these solutions are detailed on Github and in another article, our focus here is to illuminate on these issues and advocate for potential resolutions. It’s our belief that the Google team could enhance the testing experience for developers. Let’s delve into the details.

Problem 1. Syntax.

Both Espresso and the Compose UI testing framework suffer from verbose and complex test syntax, which can be difficult to handle in larger projects.

class ComposeTest {
    @get:Rule 
    val composeTestRule = createAndroidComposeRule<ComposeListActivity>()
    
    @Test
    fun simpleTest() {
        composeTestRule.onNode(hasText("Continue")).performClick()
        composeTestRule.onNode(hasTestTag("Welcome")).assertIsDisplayed()
    }
}

This is especially evident when dealing with lists, where one must account for scrolling and the absence of required list items in the initial view.

@Test
fun composeListTest() {
    composeTestRule.onNode(hasTestTag("contactsListTestTag"))
        .performScrollToNode(hasText("ContactName"))
        .onChildren()
        .filterToOne(hasText("ContactName"))
        .performClick() 
}

In small projects with just a few tests, we can manage, but in large projects with hundreds or thousands of tests, this becomes an overwhelming headache.

Solution

The solution lies in adopting the Ultron method to create ComposeRule.

class UltronComposeTest {
    @get:Rule 
    val composeTestRule = createUltronComposeRule<ComposeActivity>()
    ...
}

It retains the rule reference in the background, avoiding the need for manual management. Simply specify how to locate the UI element and initiate the interaction.

class UltronComposeTest {
    @get:Rule 
    val composeTestRule = createUltronComposeRule<ComposeActivity>()

    @Test
    fun simpleTest() {
        hasText("Continue").click()
        hasTestTag("Welcome").assertIsDisplayed()
    }
}

We will take a look at list interactions further.

Problem 2. God object.

A new problem introduced by Google is the “God object” issue, where the entire testing application API is encapsulated within the ComposeRule object. This well-known anti-pattern creates many problems. In our case, it complicates management and the extraction of interaction logic.

@Test
fun composeGodObjectTest() {
    ChatScreen(composeTestRule).sendMessage("text to send")
}

class ChatScreen(composeRule: SemanticsNodeInteractionsProvider){
    val textInput = composeRule.onNodeWithTag("message_text")
    val sendMessageButton = composeRule.onNodeWithTag("send_button")

    fun sendMessage(text: String){
        textInput.performTextReplacement(text)
        sendMessageButton.performClick()
    }
}

Solution

With Ultron, the need to manage ComposeRule is eliminated. Instead, the framework provides an API for interacting with UI elements in ANY class or method within your framework.

@Test
fun ultronTest() {
    ChatScreen.sendMessage("text to send")
}

object ChatScreen {
    val textInput = hasTestTag("message_text")
    val sendMessageButton = hasTestTag("send_button")

    fun sendMessage(text: String){
        textInput.replaceText(text)
        sendMessageButton.click()
    }
}

Problem 3. Waiters.

Like Espresso, the Compose UI testing framework does not implicitly wait for elements to appear on the screen. Instead, developers must explicitly specify in the tests that they are waiting for a certain condition or use IdlingResources. This adds significant overhead to the test implementation process, diverting attention from the test logic. Here is an example from Google samples:

@Test
fun googleTest() {
    composeTestRule.waitUntil(timeoutMillis = 10_000) {
        composeTestRule
            .onAllNodesWithContentDescription("emoji_selector_desc")
            .fetchSemanticsNodes().isEmpty()
    }
    composeTestRule
        .onNodeWithContentDescription("emoji_selector_desc")
        .assertDoesNotExist()
}

Solution

Modern frameworks, like Ultron, handle waiting logic implicitly. They automatically repeat the desired interaction until it executes successfully within a defined timeout period. You no longer need to worry about UI elements appearing with delays on the screen.

@Test
fun ultronTest() {
    hasContentDescription("emoji_selector_desc").assertDoesNotExist()
}

This approach simplifies test implementation and improves readability, allowing developers to focus more on the test logic itself and protecting them from mistakes.

Problem 4. Interaction with lists.

Lists are perhaps one of the most common elements of applications, and they often change during the applications lifecycle. We need the ability to interact with them easily and effectively, including searching for elements, describing child elements of the list item, and checking the absence of elements. Unfortunately, this is the Achilles’ heel of Google’s testing frameworks. We have big probles with RecyclerView testing and the same with Compose LazyColumn/LazyRow.

For example, consider the following test scenario where we need to validate contact’s name and status text:

Compose list item children

Here’s how this test could look with the Google framework. You don’t necessarily need to understand the following code. Simply imagine having dozens of similar tests in your project.

fun getItem(name: String): SemanticsNodeInteraction {
    return composeTestRule.onNodeWithTag("contactsListTestTag", useUnmergedTree = true)
        .performScrollToNode(hasAnyDescendant(hasText(name)))
        .onChildren()
        .filterToOne(hasAnyDescendant(hasText(name)))
}

@Test
fun assertItemChildrenContentTest() {
    getItem("Ross Geller")
        .onChildren()
        .filterToOne(hasTestTag("nameTestTag"))
        .assertTextEquals("Ross Geller")
    getItem("Ross Geller")
        .onChildren()
        .filterToOne(hasTestTag("statusTestTag"))
        .assertTextEquals("UNAGI")
}

The Compose UI testing framework lacks the capability to search for child elements in the list beyond the first generation, forcing heavy reliance on the current structure of list items. Any changes to the item structure would require rewriting numerous tests, leading to maintenance challenges.

Solution

Ultron simplifies interaction with lists by handling scrolling and the absence of elements seamlessly.

val contactsList = composeList(hasTestTag("contactsListTestTag"))

@Test
fun ultronListTest() {
    contactsList.item(hasText("Ross Geller")).click()
}

The framework allows you to abstract away the structure of your list items, enabling deep searching for child elements in any structure.

class ContactItem : UltronComposeListItem() {
    val name by child { hasTestTag("nameTestTag") }
    val status by child { hasTestTag("statusTestTag") }
}

It’s recommended to encapsulate item search logic into a separate method.

fun getItem(name: String): ContactItem {
    return contactsList.getItem(hasText(name))
}

Use the new method to interact with the item and its children.

@Test
fun ultronListTest() {
    getItem("Ross Geller").apply {
        name.assertTextEquals("Ross Geller")
        status.assertTextEquals("UNAGI")
    }
}

This greatly reduces the number of maintenance issues for your tests.

Problem 5. Lack of patterns for automated tests.

All examples of test implementation from the Google team are limited to simple applications with a couple of buttons and a simple list. You can see them here. The following code is just a sample; you don’t need to understand it.

class NavigationTest {
    @Test
    fun profileScreen_back_conversationScreen() {
        val navController = getNavController()
        // Navigate to profile        \
        navigateToProfile("Taylor Brooks")
        // Check profile is displayed
        assertEquals(navController.currentDestination?.id, R.id.nav_profile)
        // Extra UI check
        composeTestRule
            .onNodeWithText("display_name")
            .assertIsDisplayed()
    
        // Press back
        Espresso.pressBack()
    
        // Check that we're home
        assertEquals(navController.currentDestination?.id, R.id.nav_home)
    }
    ...
    private fun navigateToProfile(name: String) {
        composeTestRule.onNodeWithContentDescription("navigation_drawer_open")
            .performClick()
        composeTestRule.onNode(hasText(name) and isInDrawer()).performClick()
    }
}

The test logic is encapsulated within methods in the test class, lacking a clear architectural pattern. The community has realised that using such simple patterns as Page Object and Steps significantly facilitates the process of maintaining automated tests.

Solution

A solid test architecture simplifies and maintains tests. By organising your test framework into layers you can follow the separation of concerns principle.

Ultron recommended UI tests architecture

Ultron comes with recommended approach, base classes for Page Object pattern implementation. All features of the framework are consistent with this architecture. To demonstrate this lets refactor our list interaction test.

object ContactsListScreen : Screen<ContactsListScreen>(){
    private val contactsList = composeList(hasTestTag(contactsListTestTag))

    private class ContactItem : UltronComposeListItem() {
        val name by child { hasTestTag(contactNameTestTag) }
        val status by child { hasTestTag(contactStatusTestTag) }
    }

    fun getItem(name: String): ContactItem {
        return contactsList.getItem(hasText(name))
    }
}

@Test
fun ultronListItemTest() {
    ContactsListScreen.getItem("Ross Geller").apply {
        name.assertTextEquals("Ross Geller")
        status.assertTextEquals("UNAGI")
    }
}

ContactsListScreen is essentially an implementation of the Page Object pattern. It stores the description of UI elements.

ContactItem describes list item children. Once we need to modify it, we have a single point of truth.

getItem() method allows us to incapsulate the logic of item search, providing us a great maintainability for this UI element modification.

Problem 6. Lack of understandable reporting.

One major pain point in the testing process is the absence of a robust reporting system that provides clear insights into the state of the test run along with necessary artefacts such as test logs and screenshots corresponding to different stages of test execution. The standard JUnit reports fall short in providing comprehensive information, especially in large projects, causing significant headaches for developers.

Solution

Ultron addresses this issue by automatically generating artefacts for Allure reports. For those unfamiliar with Allure, I highly recommend exploring its capabilities. Allure reports offer detailed and visually appealing insights into test execution, including comprehensive logs and screenshots, empowering developers to efficiently analyze test results and troubleshoot issues.

Here is an example of a failed test report from the Ultron framework. You can find detailed step info, attached screenshots, full logcat logs, the framework interactions log, and window hierarchy dump.

Summary

In essence, while this article highlights some testing challenges with Jetpack Compose, it only scratches the surface. Deeper issues remain unexplored. Hopefully, community demands for improved testing will drive Google to enhance the framework. In the meantime, relying on third-party solutions like Ultron is crucial.