Ultron — simple, stable, maintainable Android UI tests

Dear reader!

Android UI tests development can be challenging. Over the past decade, during my extensive involvement in Android tests automation, I have encountered numerous difficulties. In this article, I will clarify the primary issues and propose optimal solutions based on my experience.

For those less familiar with Android automation, long story short. Android UI can be built using two approaches: the traditional View components (old approach) and the modern Jetpack Compose toolkit. Google offers three frameworks for creating UI tests: Espresso for View-based UI, Compose testing framework for Compose UI, and UI Automator, mostly used for testing app integration.

When examining the Espresso test code, I asked myself why it’s so complex. Let’s explore an example from the official documentation:

@Test
fun greeterSaysHello() {
    onView(withId(R.id.name_field)).perform(typeText("Steve"))
    onView(withId(R.id.greet_button)).perform(click())
    onView(withText("Hello Steve!")).check(matches(isDisplayed()))
}

• withId(R.id.name_field) — the definition of how we search for the UI element
• typeText("Steve") , click()— denotes the actions to be performed on the designated elements
• isDisplayed() — the assertion we wish to execute

Upon deeper reflection, I realized that we could eliminate redundant code like onView() , perform() and check().

@Test
fun greeterSaysHello() {
    withId(R.id.name_field).typeText("Steve")
    withId(R.id.greet_button).click()
    withText("Hello Steve!").isDisplayed()
}

In my estimation, this modified version better describes the test intentions. Imagine extrapolating such efficient tests into a real project with hundreds of test cases. The efficiency gained by these simple improvements would be significant. Furthermore, you can already write tests using this modern approach and enhance it even further by leveraging an open-source framework!

Over the past four years, I have developed Ultron — a solution that streamlines test automation. It has gained traction among enthusiasts, and numerous companies successfully utilize it, running hundreds and thousands of UI tests on each merge request. Embracing Ultron will not only save you time but also enhance your test development experience.

This is my very first article on Medium :) In this series of articles, I’ll introduce Ultron’s abilities and share my experience in Android test automation.

Intro

Ultron is built upon the Espresso, UI Automator, and Compose UI testing frameworks. When creating the framework, I aimed for three main principles:

1. Simplicity. Making tests easy to understand, ensuring the code’s purpose is clear at a glance. This minimizes time and mental effort in test development.
2. Stability. Ensuring the reliability of test execution and addressing the issue of flaky tests, liberating developers to focus on test logic instead of wrestling with test stability.
3. Maintainability. Improving code maintainability and customization allows for quick diagnosis of failures and modification of tests to fit specific needs.

In the forthcoming sections, I will expound on how each of these principles materializes in the framework.

Simplicity

Ultron vs Google Frameworks

Now, let us delve into a comparison between the syntax of Ultron and Google’s frameworks. It becomes evident that Ultron’s syntax is considerably more straightforward and accessible. Let us examine some illustrative samples:

Simple Compose operation

// Compose framework
class ComposeFrameworkTest {
    @get:Rule
    val composeTestRule = createComposeRule<YourActivity>()
    @Test
    fun myTest() {
        composeTestRule.onNode(hasTestTag("Continue")).performClick()
        composeTestRule.onNodeWithText("Welcome").assertIsDisplayed()
    }
}

// Ultron framework
class UltronComposeTest {
    @get:Rule
    val composeTestRule = createUltronComposeRule<YourActivity>()
    @Test
    fun myTest() {
        hasTestTag("Continue").click()
        hasText("Welcome").assertIsDisplayed()
    }
}

The Google Compose testing API is provided by the AndroidComposeTestRule object. However, it is worth noting that this approach may not be the most optimal.

In contrast, Ultron dispenses with the need for the AndroidComposeTestRule object to execute any operation. Simply create the rule using the framework’s static method createUltronComposeRule, and voilà! You gain the ability to perform stable compose operations in ANY class. Create a SemanticsMatcher, such as hasTestTag("Continue") and call the desired operation.

The SemanticsMatcher is a crucial component of the Android Compose testing framework, enabling the identification of target nodes for interaction. Refer to the cheatsheet for a comprehensive list of available matchers.

Further articles will describe the Compose part of Ultron. For now, you can find more information about it in documentation.

Compose list operation

// Compose framework
val itemMatcher = hasText(contact.name)
composeRule
    .onNodeWithTag(contactsListTestTag)
    .performScrollToNode(itemMatcher)
    .onChildren()
    .filterToOne(itemMatcher)
    .assertIsDisplayed()

// Ultron
composeList(hasTestTag(contactsListTestTag))
    .item(hasText(contact.name))
    .assertIsDisplayed()

Interacting with lists in Compose can be quite cumbersome. Ultron, however, simplifies this complexity by providing a convenient method called composeList(), which reveals the magic API for interacting with list items.

Simple Espresso assertion and action

// Espresso
onView(withId(R.id.send_button))
  .check(isDisplayed())
  .perform(click())

// Ultron
withId(R.id.send_button)
  .isDisplayed()
  .click()

Names of all of the framework operations are the same as Espresso. It also introduces a set of additional operations.

Action on RecyclerView list item

// Espresso
onView(withId(R.id.recycler_friends))
    .perform(
        RecyclerViewActions
            .actionOnItem<RecyclerView.ViewHolder>(
                hasDescendant(withText("Janice")),
                click()
            )
        )

// Ultron
withRecyclerView(R.id.recycler_friends)
    .item(hasDescendant(withText("Janice")))
    .click()

RecyclerView has been a longstanding challenge in Android automation. Ultron rises to this challenge and offers an elegantly streamlined approach to interact with RecyclerView.

Espresso WebView operation

// Espresso
onWebView()
    .withElement(findElement(Locator.ID, "text_input"))
    .perform(webKeys("some title"))
    .withElement(findElement(Locator.ID, "button1"))
    .perform(webClick())
    .withElement(findElement(Locator.ID, "title"))
    .check(webMatches(getText(), containsString("some title")))

// Ultron
id("text_input").webKeys("some title")
id("button1").webClick()
id("title").hasText("some title")

Espresso Web part syntax is even more complex, diverting your focus away from test logic. With Utron, you can effortlessly create a web element using one of factory methods like id("text_input") and execute the required action or assertion.

UI Automator operations

Similar considerations apply to UI Automator operations.

// Espresso
val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
device
    .findObject(By.res("com.atiurin.sampleapp:id", "button1"))
    .click()

// Ultron
byResId(R.id.button1).click()

Ultron simplifies UI elements creation with a set of factory methods, such as byResId(R.id.button1), unlocking the door to a magical API.

How to use Ultron in 3 simple steps.

I strive to promote the proper construction of the test framework architecture, delineation of responsibilities across different layers, and other essential practices. As a result, I offer a recommended approach to test development using the framework.

The methodology is described using the Espresso part, yet it can similarly be implemented for Compose and UI Automator.

Step 1. Create a PageObject/ScreenObject.

It’s recommended to structure your tests using the Page Object pattern. Create a Page Object for each screen of your app and specify the screen’s UI elements as Matcher<View> objects.

object ChatPage : Page<ChatPage>() {
    private val messagesList = withId(R.id.messages_list)
    private val clearHistoryBtn = withText("Clear history")
    private val inputMessageText = withId(R.id.message_input_text)
    private val sendMessageBtn = withId(R.id.send_button)
}

By encapsulating UI elements within a Page Object, you can easily manage and reuse them in your code. You might wonder why the Page Object is an object instead of a class; this is because I strongly recommend maintaining a stateless Page Object.

If you prefer using the term Screen instead of Page, you have the option to express the same code in an alternative manner:

object ChatScreen : Screen<ChatScreen>() {...}

Step 2. Describe user step methods in Page Object

Within your Page Object, define user step methods that encapsulate common interactions with the screen. These methods provide a higher level of abstraction and make your tests more readable.

object ChatPage : Page<ChatPage>() {
    fun sendMessage(text: String) = apply {
        inputMessageText.typeText(text)
        sendMessageBtn.click()
        getMessageListItem(text).text
             .isDisplayed()
             .hasText(text)
    }

    fun clearHistory() = apply {
        openContextualActionModeOverflowMenu()
        clearHistoryBtn.click()
    }
}

By utilizing user step methods, you can construct complex test scenarios in a concise and expressive manner.

Step 3. Call user steps in Test

In your test methods, call the user step methods defined in the corresponding Page Object. This approach separates the test logic from the UI interaction, improving test readability and maintainability.

@Test
fun friendsItemCheck() {
    FriendsListPage {
        assertName("Janice")
        assertStatus("Janice", "Oh. My. God")
    }
}

@Test
fun sendMessage() {
    FriendsListPage.openChat("Janice")
    ChatPage {
        clearHistory()
        sendMessage("test message")
    }
}

By following these steps, you can structure your test framework with a clear separation of responsibilities and achieve a more maintainable and scalable test suite.

In general, it all comes down to the fact that the architecture of your project will look like this:

• Only the TEST CLASS is allowed to manage TEST DATA and provide it for actions and assertions.
• The PAGE OBJECT contains private definitions of APP UI ELEMENTS, preventing direct usage in the TEST CLASS. Instead, it presents public STEPS, that interact with APP UI ELEMENTS using TEST ADAPTERS.
• TEST ADAPTERS act as mechanisms providing the interface to interact with your application. Initially, this role was fulfilled by Espresso, UI Automator, and Compose testing frameworks, but now Ultron takes the lead.

Stability

One of the key strengths of the framework is the stability it brings to your tests. Several features contribute to making your tests stable.

Exceptions handling

The framework catches specified exceptions and automatically retries operations during a timeout (default is 5 seconds). This eliminates the need for manual waiting using methods like sleep() and avoids unnecessary code clutter. Instead, you can concentrate on testing the desired scenario. Moreover, you have the flexibility to customize the list of exceptions that are handled and set custom timeouts for specific operations. This allows for fine-tuning and tailoring the behavior of the framework to meet your specific requirements. Here’s an example:

withId(R.id.result).withTimeout(10_000).hasText("Passed")

Customizable operation result processing

The framework enables you to process the result of any operation in a custom way. It provides comprehensive information for this purpose. There is a method that opens the door: withResultHandler

This method provides a list of exceptions, with the last one being the exception that was thrown. This functionality becomes particularly useful when interacting with custom Views that might throw unusual exceptions, such as those involving media players. It’s also very useful while debugging your tests.

player.withResultHandler { operationResult ->
    val exception = operationResult.exceptions.last()
    if (exception is PlayerDisabledException){
        //enable player 
    }
}.click()

Boolean operation result

You can retrieve the result of any operation as a boolean value. For instance:

val isButtonDisplayed = withId(R.id.button).isSuccess { isDisplayed() }
if (isButtonDisplayed) {
    //do some reasonable actions
}

I don’t really like this feature. If we want to use it, that means we don’t fully control our test preconditions. However, I acknowledge within the complicated Android world we can’t control everything or it might be too much difficult.

Custom assertion of any action

This feature helps you to avoid unstable behavior or freezes in your application. For instance, if you need to assert that an element appears after a click, and if it doesn’t, you need to repeat the click action:

button.withAssertion("Assert title is displayed") {
    title.isDisplayed()
    // here could be some custom assertion of app condition
}.click()

You can also customize the duration of the assertion using the withTimeout() method.

button.withAssertion("Assert title is displayed") {
    title.withTimeout(3_000L).isDisplayed()
}.click()

Incredible interaction with RecyclerView and Compose lists.

To avoid redundancy, I won’t replicate the documentation here. It’s essential to carefully read about RecyclerView and Compose LazyColumn/LazyRow interactions. Key features of interaction with lists include:

• Automatic scrolling to items
• Simple searching for unique items

recycler.item(position = 10).click() // find item at position 10 and scroll to this item 
recycler.item(matcher = hasDescendant(withText("Janice"))).isDisplayed()

• Interacting with the first and last items

recycler.firstItem().click() //take first RecyclerView item
recycler.lastItem().isCompletelyDisplayed()

• Searching for non-unique items using matching criteria

// if it's impossible to specify unique matcher for target item
val matcher = hasDescendant(withText("Friend"))
recycler.itemMatched(matcher, index = 2).click() //return 3rd matched item, because index starts from zero
recycler.firstItemMatched(matcher).isDisplayed()
recycler.lastItemMatched(matcher).isDisplayed()
recycler.getItemsAdapterPositionList(matcher) // return positions of all matched items

• Interacting with child elements of list items

Just describe the class that tells the framework how to find a child element inside the RecyclerView item.

class FriendRecyclerItem : UltronRecyclerViewItem() {
    val avatar by lazy { getChild(withId(R.id.avatar)) }
    val name by lazy { getChild(withId(R.id.tv_name)) }
    val status by lazy { getChild(withId(R.id.tv_status)) }
}

Interact with elements

recycler.getFirstItem<FriendRecyclerItem>().status.hasText("UNAGI")
recycler.getItem<FriendRecyclerItem>(matcher = hasDescendant(withText("Janice")))
        .status.textContains("Oh. My. God")

All of these RecyclerView features also apply to Compose lists.

Ability to bypass Espresso Idling mechanism

If you’ve spent time in Android automation world, you’ve likely encountered the challenge of frozen tests. This often results from the Espresso Idling mechanism, which determines whether the application is idle. Animation, particularly custom app animation, is a common factor contributing to Espresso freezes.

While the optimal solution would be to disable these animations, it’s not always possible. In such cases, a trade-off is inevitable. Here’s a method that empowers you to interact directly with any View without waiting for Espresso:

withId(R.id.video_player).performOnViewForcibly {
    val playerView = this as VideoPlayer
    playerView.performClick() //direct interaction with custom view object
    assertTrue(playerView.isPlayed) //isPlayed - custom public property of VideoPlayer View
}

Maintainability

The framework incorporates several features that contribute to exceptional maintainability for your UI tests.

Configuration

The most straightforward approach to setting up the framework is by utilizing the recommended configuration:

@BeforeClass @JvmStatic
fun setConfig() {
    UltronConfig.applyRecommended()
    UltronAllureConfig.applyRecommended()
    UltronComposeConfig.applyRecommended()
}

Moreover, Ultron offers the capability for flexible configuration of your test framework. You can customize various aspects such as:

• Different timeouts for operations, RecyclerView loading, etc.
• Report artifacts and policies for their attachment.
• List item search limit for RecyclerView and Compose lists.
• Logging options, among other options.

UltronConfig.apply {
    logToFile = true
    accelerateUiAutomator = true
    operationTimeoutMs = 10_000
}
UltronAllureConfig.apply {
    attachLogcat = false
    attachUltronLog = true
    ...
}
UltronComposeConfig.apply {
    lazyColumnItemSearchLimit = 100
    ...
}

Extensions

Ultron leverages the power of Kotlin extension functions. This Kotlin feature allows the framework to provide such a simple API and hide complex processes.

You can extend the framework by using its native approach along with your custom operations. For example, let’s add a new Espresso action. All Espresso Ultron operations are described in UltronEspressoInteraction class. That is why you need to extend this class, like this:

fun <T> UltronEspressoInteraction<T>.appendText(value: String) = 
    perform { _, view ->
        val textView = (view as TextView)
        textView.text = "${textView.text}$value"
    }

Create an extension function for Matcher<View>

fun Matcher<View>.appendText(text: String) = 
    UltronEspressoInteraction(onView(this)).appendText(text)

You can use the new action anywhere

withId(R.id.text_input).appendText("some text to append")

Here’s an additional illustration showcasing the simplicity of expanding the Compose part of the framework. Consider a scenario where we aim to retrieve the width of the SemanticsNode.

fun UltronComposeSemanticsNodeInteraction.getWidth(): Int = execute {
    semanticsNodeInteraction.fetchSemanticsNode().size.width
}

fun SemanticsMatcher.getWidth(): Int = 
    UltronComposeSemanticsNodeInteraction(this).getWidth()

// use it in @Test or Page Object
val buttonWidth = hasTestTag("Button").getWidth()

Examples of every component extension can be located within the framework documentation.

Test conditions management

Preparing data and application conditions ready for Android UI tests is unquestionably a challenging task. While it might seem manageable with a small number of tests, the complexity escalates exponentially when dealing with a larger test suite consisting of hundreds of tests.

The Android JUnit 4 testing framework struggles to effectively handle the complexities of test data preparation. However, Ultron steps in to offer an exceptional solution for managing test conditions. This remarkable feature empowers you to:

• Order rules execution with RuleSequence, which is an amazing replacement for JUnit 4’s RuleChain.
• Avoid usage of @Before and @After methods by replacing them with a lambda.
• Define specific conditions for a single test, a group of tests, and all tests.
• Group several different conditions for a single test and manage the order of their execution.

I think it would be beneficial to explain this in its own article. I’ll work on it if there’s interest. You can find more details in the documentation.

Listeners

Ultron allows you to listen to all stages of operation execution.

abstract class UltronLifecycleListener {
    // executed before any action or assertion
    override fun before(operation: Operation) = Unit

    // executed when action or assertion failed
    override fun afterFailure(operationResult: OperationResult<Operation>) = Unit

    // executed when action or assertion has been executed successfully
    override fun afterSuccess(operationResult: OperationResult<Operation>) = Unit

    // executed in any case of action or assertion result
    override fun after(operationResult: OperationResult<Operation>) = Unit    
}

The Operation object contains operation details like name, description, type, and timeout. Similarly, the OperationResult object holds its outcome, including success status, exceptions, description, and references the corresponding Operation.

All listener methods are invoked prior to any exceptions being thrown. This ensures that any exceptions occurring within your tests will be managed as per your specified criteria.

To add your own listener to the framework:

1. Create a subclass of UltronLifecycleListener.

class MyCustomLifecycleListener : UltronLifecycleListener(){ .. }

2. Add the new listener to the framework config.

abstract class BaseTest {
    companion object {
        @BeforeClass @JvmStatic
        fun configureUltron() {
            UltronConfig.addGlobalListener(MyCustomLifecycleListener())
        }
    }
}

The listeners mechanism offers remarkable flexibility. It is possible to exclude certain operations from listening. The listener could be applied for Espresso and switched off for Compose, etc.

Exception description

It’s a significant problem of the default Android test. Once it fails, you have to spend too much time understanding what’s going on. My favorite exceptions are NoMatchingViewException and AmbiguousViewMatcherException . Once you encounter one of these exceptions you have to spend minutes to find meaningful info by scrolling over the full window hierarchy. Additionally, it’s common for the absence of a stack trace due to the excessive length of the error message. This can make identifying the specific piece of code that triggers the exception quite challenging.

Ultron addresses this issue by offering detailed exception information. For instance, this test:

@Test
fun click_notExisted() {
    withText("Not existed element").withTimeout(1000).click()
}

throws the following exception message:

This message precisely describes the reason for the failure and the duration the framework attempted to repeat the action. And, of course, it provides you with the reference to the piece of code that threw the exception.

Allure report

Ultron facilitates test reporting by integrating Allure reports. By following the recommended configuration and specifying the test instrumentation runner, you can effortlessly generate Allure artifacts. The Allure report provides a comprehensive overview of your test results, including detailed step-by-step information, logs, screenshots and other meaningfull artifacts. Whether you’re using traditional Espresso or embracing Compose, Ultron’s Allure integration ensures your test reports are informative and visually appealing.

Set testIntrumentationRunner (example build.gradle.kts).

android {
  defaultConfig {
    testInstrumentationRunner = "com.atiurin.ultron.allure.UltronAllureTestRunner"
    ...
  }

and apply the recommended config in your BaseTest class (example BaseTest).

@BeforeClass @JvmStatic
fun setConfig() {
    UltronConfig.applyRecommended()
    UltronAllureConfig.applyRecommended()
}

For instance, this test:

@Test
fun specialFailedTest() {
    val firstMessage = "first message"
    val secondMessage = "second message"
    FriendsListPage.openChat("Janice")
    ChatPage {
        clearHistory()
        sendMessage(firstMessage)
        sendMessage(secondMessage)
        // position starts from 0
        assertMessageTextAtPosition(1, firstMessage)
    }
}

can generate the following report:

The generated report provides comprehensive details to identify the cause of failure, including screenshots, window hierarchy, logcat log, and a complete interaction log.

Compose Allure report

For Compose add 4 lines more to the configuration method

@BeforeClass @JvmStatic
fun setConfig() {
    ...
    UltronComposeConfig.applyRecommended() 
    UltronComposeConfig.addListener(ScreenshotAttachListener())
    UltronComposeConfig.addListener(WindowHierarchyAttachListener())
    UltronComposeConfig.addListener(DetailedOperationAllureListener())
}

Simple Compose test generates the detailed report out of the box

@Test
fun testTagTest() {
    hasText("Like count = 0").click()
    hasTestTag("LikeS_CounteR").assertTextEquals("some invalid text")
}

Even in default configuration the framework provides detailed info for Allure report.

Conclusion

Over the years, I’ve come a long way, developing automated tests using various frameworks and encountering countless challenges along the way. You don’t have to tread the same path as I did; instead, you can embark on your journey using my experience as a guide. By adopting the recommended approach and leveraging Ultron’s powerful features, you can enjoy the devopment of simple, stable and maintainable tests.

So go ahead, embrace Ultron, and let your UI tests conquer the Android universe!