Creating Fast, Reliable, Focused UI Automation With Atomic Tests

Nikolay Advolodkin
Mar 4 · 9 min read

What is an atomic automated test?

Your automated test should form a single irreducible unit. This means that your tests should be extremely focused and test only a single thing. A single automated test should not do something like end-to-end automation.

An atomic test is one that tests only a single feature. You can usually tell that a test is atomic when:

  • The test will only have one assertion or two assertions at most. Because sometimes we need one assertion to make sure our state is correct
  • Atomic tests have very few UI interactions and they’re only on a maximum of two screens. In rare cases, an atomic test might navigate through 3 screens (although I’d like to see this example)

Problems solved by atomic tests

  • Unreliable automated test results
  • Slow tests
  • Tests that are hard to debug
  • Tests that don’t provide the correct point of failure

A good rule of thumb that I use on my teams is:

Automated acceptance test should not run longer than 1 minute on your local resources

If your test runs longer than 1 minute, then you should keep reading on why that might be dangerous. The advantages of writing atomic tests are numerous…

Here are some examples of atomic GUI tests:

500% Reduction in Test Suite Execution Time (Case Study)

In a recent case study, we found that 18 long end to end tests ran in ~20 min. Using a much larger suite of 180 atomic tests, with the same exact code coverage, running in parallel, we were able to decrease the entire suite execution time to ~4min.

Advantages of atomic tests

1. Atomic tests fail fast

First, writing atomic tests allows you to fail fast and fail early. This implies that you will get extremely fast and focused feedback. If you want to check the state of a feature, it will take you no longer than 1 minute.

2. Atomic tests decrease flaky behaviour

Tests that complete in 2 minutes or less are twice as likely to pass as tests lasting longer than two minutes. In other words, the longer a test takes to run, the more likely it is to fail.

https://info.saucelabs.com/sauce-labs-continuous-testing-benchmark-report-M.html

Second, writing atomic tests reduces flakiness because it decreases the number of possible breaking points in that test. Flakiness is less of a problem with unit or integration tests. But it is a large problem with acceptance UI automation.

Here’s an example:

  1. Open UltimateQA.com home page
  2. Assert that the page opened
  3. Assert that each section on the page exists
  4. Open Blog page
  5. Search for an article
  6. Assert that the article exists

For UI automation, every single step is a chance for something to go wrong…

A locator may have changed, the interaction mechanism may have changed, your synchronization strategy may be broken, and so on.

Therefore, the more steps that you add, the more likely your test is to break and convey false positives.

3. Atomic checks allow for better testing

The third benefit of writing atomic tests is that if it fails, it will not block other functionality from being tested. For example, the test I mentioned above. If it fails on Step 3, then you might never get to check if the Blog page works or that the Search functionality works. Assuming that you don’t have other tests to check this functionality. As a result of a large test, you will reduce your test coverage.

4. Atomic tests are short and fast

Finally, another great benefit of writing atomic tests is that they will run quicker when parallelized…

How I got a 98% performance enhancement in test execution speed with a single change?

In the scenario above, I had a suite of 18 end-to-end tests that were NOT atomic and were not running in parallel.

Maintaining the same code coverage, I broke down my tests into 180 tiny, atomic tests…

Ran them in parallel and decreased the average time of test case to 1.76s from 86s.

Your entire automation suite run time will be as fast as your slowest test

Nikolay Advolodkin

If you want to learn more of the automation best practices as well CI/CD and framework development then check out The Complete Selenium WebDriver with Java Bootcamp.

By the way, I have seen automated tests that take 30–90 minutes to execute. These tests are extremely annoying to run because they take so long. Even worse, I’ve never seen such a test produce valuable feedback in my entire career. Only false positives.

Case Study of Amazing Automation:

Break down:

30,000 tests/week

96% pass rate

Average test run time: 30 sec (In the cloud!)

The average number of Selenium commands per test: < 30

Ummm, yea. That’s what atomic tests will do 🔥🔥👏👏

How to break up giant end-to-end UI tests?

Okay, you, believe me, atomic tests are good.

But how can you break up your large end-to-end tests, right?

Trust me, you’re not the only one struggling with this situation…

It gets worse:

Daily, I encounter clients that have the same issue.

Furthermore, I wish that I could provide a simple answer to this. But I cannot…

For most individuals, this challenge is one of technology and culture.

However, I will provide a step by step guide to help you have atomic tests.

It won’t be easy… But when you achieve it, it will be SO Sweet!

Here is a simple scenario:

  1. Open Amazon.com
  2. Assert that the page opens
  3. Search for an item
  4. Assert that item is found
  5. Add item to cart
  6. Assert that item is added
  7. Checkout
  8. Assert that checkout is complete

The first problem is that many automation engineers assume that you must do an entire end-to-end flow for this automated test.

You must complete step 1 before step 2 and so on… Because how can you get to the checkout process without having an item in the cart?

The automated testing best practices approach is to be able to inject data to populate the state of the application before any UI interactions.

Side Note:

If you would like to see a live code demonstration, I tackle that in this video at ~13min:

How to break up e2e tests into atomic tests

How to manipulate test data for UI automation?

You can inject data via several options:

  1. Using something like a RESTful API to set the application into a specific state
  2. Using JavaScript
  3. Injecting data into the DB to set the application in a certain state
  4. Using cookies

If you can inject data between the seams of the application, then you can isolate each step and test it on its own

Some Options:

  1. Use an API to send a web request that will generate a user
  2. Use an API that will generate an item in your Amazon cart
  3. Now you can pull up the UI to the cart page and checkout using web automation
  4. Clean up all test data after

This is the best practice approach. You tested the checkout process without using the UI for steps one through three.

Using an API is extremely fast… A web request can execute in like 100 ms.

This means that steps 1,2,4 can take less than one second to execute. The only other step you will need to do is to finish the checkout process.

It gets better:

Using an API is much more robust than using a UI for test steps. As a result, you will drastically decrease test flakiness in your UI automation.

If you want to learn more of the automation best practices as well CI/CD and framework development then check out The Complete Selenium WebDriver with Java Bootcamp.

How To Control App State Using JavaScript?

Probably the most common impediment to atomic testing is the login screen. And most of our apps have one.

So how do we remove this from our test so that our test can be atomic?

Here’s one example:

1. We execute some JavaScript with our automation framework like this:

Congratulations, we are now logged in 🙂

Now use your UI testing tool to perform the single operation that you want to test.

Here’s how a full atomic test would look like:

Notice how the test only has one UI action and one assertion…

That’s a sign of an atomic test in UI automation.

Here’s a tutorial that shows you how to create an atomic test by controlling the state of the UI using JS.

How to bypass a login mechanism using JavaScript injection

How To Coordinate API and UI Interactions In One Test?

What if you wanted to coordinate API and UI actions in a single test? Well, that would look something like this.

The most common situation is not to alternate Selenium and API calls. Normally we do API stuff to set upstate. UI stuff to assert functionality. API stuff to clean up. It’s not common to do Selenium, API, Selenium, API, Selenium and so on.

However, it’s possible for a user case for sure.

Just remember that Selenium or any other UI framework has nothing to do with your App API. A UI automation library is all about the front-end, the UI. The API part of your test is all about managing state and test data, the back end.

What if you can’t inject data for testing?

I know that the world isn’t perfect and many of us aren’t lucky enough to have applications that are developed with testability in mind.

So what can you do?

You have two options:

1. Work with Developers to make the application more testable

Yes, you should work with the developers to make your application more testable. Not being able to easily test your application is a sign of poor development practices.

This does mean that you will need to leave your cube and communicate across silos to break down communication barriers.

Frankly, this is part of your job. You need to communicate across teams and work together to create a stable product.

If the product fails, the whole team fails, not just a specific group

Again, it’s not easy…

I’ve worked at one company where it took me two years to simply integrate Developers, automation, and manual QA into a single CI pipeline.

It was a weekly grind to have everyone caring about the outcome of the automation suite.

And in the end, our team was stronger and more agile than ever.

Trust me, this is doable and most developers are happy to help. But you must be willing to break down these barriers.

Here’s the second option, and you won’t want to hear it:

2. If your application is not automation-friendly, don’t automate

If you can’t work with the developers because you’re unwilling…

Or if the company culture doesn’t allow you to…

Then just don’t automate something that won’t provide value. I know that your manager asked you to automate it…

However, we are the automation engineers. We are the professionals.

We must decide what to automate and not to automate based on our understanding of application requirements.

We were hired because of our technical expertise, because of our abilities to say what is possible, what is not possible, and what will help the project to succeed.

Although it might feel easy to say “yes, I will automate your 30-minute scenario”, it’s not right to do so.

If your manager is non-technical, they should not be telling you how to do your job. You don’t see managers telling developers how to code. Why is it okay for managers to tell an automation engineer what to automate?

The answer is it’s not okay.

You must be the expert and decide on the correct approach to do your job.

If you don’t agree with me…

Check out this video from Robert Martin — arguably one of the best Software Developers of this century.

Bob Martin on professionalism

He does a better job explaining professionalism than I ever could 🙂

If you want to learn more of the automation best practices as well CI/CD and framework development then check out The Complete Selenium WebDriver with Java Bootcamp.

What do you think? Can you write atomic tests in your application?

The Startup

Medium's largest active publication, followed by +605K people. Follow to join our community.

Thanks to The Startup

Nikolay Advolodkin

Written by

Nikolay Advolodkin is a self-driven SDET on a lifelong mission to create profound change in the IT world and ultimately leave a legacy for his loved ones

The Startup

Medium's largest active publication, followed by +605K people. Follow to join our community.

More From Medium

More from The Startup

More from The Startup

More from The Startup

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade