Slalom Build
Published in

Slalom Build

Don’t Automate Test Cases

How directly automating test cases leads to unwieldy and bloated automation suites that provide little or no value.

The Costs and Benefits of Test Automation

To understand why automating existing test cases is so problematic, we need to step back and review a bit of automation theory. Specifically, we need to dig into the automation costs and benefits, look at the expected value-over-time of automated tests, then look at how the expected value changes with different types of tests. We can then look at how automating test cases using a simple automation factory approach would impact overall test suites.

The most oversimplified analysis of test automation you will ever see.
  • The cleanliness of the existing test framework and suite
  • The ease of and ability to setup test state (eg test data)
  • The availability of an acceptable test oracle
  • The volatility of the interfaces or features the test will interact with
  • The stability of the environment against which the test will run
  • The technical skill of the QAs expected to create and maintain the automation
  • How long the test will be a valid check of the SUT.
  • How expensive (in time, or otherwise) it is to validate the same test manually
  • How error prone the test is to run manually

The Many Types of Automated Tests

Next, let’s consider what the value-over-time graphs would look like for different types of automated tests. By types of tests, we mean everything from small, code-level unit tests, slightly larger “social” unit tests, even larger component tests, higher level integration tests, tests that bypass the UI and hit APIs directly, tests that mock the APIs and exercise just the UI, E2E tests that span the full tech stack, etc.

  • Unit tests are (or should be) immune to external state. Meaning they set up mocks, doubles, stubs, etc. to deterministically control the execution path.
  • Unit tests can be executed in milliseconds, suites of unit tests in seconds.
  • Unit tests will likely be executed thousands of times a day, not only within a CI/CD pipeline for every commit, but also locally by each developer as code is written.
  • Even with 100% coverage, unit tests cannot prove the application actually works as expected, and only validate an incredibly small (usually singular) thing.
  • E2E tests are executed against a full environment. Often parts of this environment are shared.
  • E2E tests often include many (dozens, even hundreds) of serial steps.
  • E2E tests are the slowest of all tests, possibly running for minutes.
  • E2E tests usually have to drive functionality through a user interface.
  • E2E tests are usually executed much later in a CI/CD pipeline.
  • E2E tests are the only type of automated tests that demonstrate the application works as the customer would use it.

Where to Automate

The nature of E2E tests makes them costly to create and costly to maintain. They necessarily rely on (or could be impacted by) the most states across the most systems. They are more prone to system timing, synchronization, network, or external dependency issues. They usually drive some or all functionality through a web browser, an interface designed to be consumed by a human, not software. Because they are executed against a full environment, it is more likely that these tests will have to share part or all of this environment with other tests or users, possibly leading to collisions and unexpected results.

Many types of tests. The actual names used for each test type vary significantly between organizations.

Automation Factory and the Top Heavy Suite

Back to the automation factory!

A very hypothetical Agile user story… what type of tests would be created from this?
All new functionality is tested with dev unit tests and new E2E tests, leading to a top-heavy and hard to maintain test suite!

Symptoms of the Automation Factory

Using the automation factory approach of test-cases-in, automated-test-cases-out automation approach tends to lead to some very problematic but unfortunately common symptoms:

  • Automation teams whose sole purpose is to investigate then triage the failures of the previous execution—which consumes most of their time
  • Test failures where the accepted mitigation is simply to rerun the test until it works
  • The removal of the suite from the CI/CD pipeline, or demoting it to a non-blocking step
  • Test suites where developers avoid or outright refuse to run them because they don’t trust the results
  • Suites with thousands of tests, spread across hundreds of folders (or even different repos!), with duplicated tests, commented tests, and tests nobody knows what they do or how they got there
  • Herculean efforts by automation engineers to manage the bloated suite of tests, or to hide the complexity behind a layer of gherkin (eg: Cucumber, etc.)

Healthy Automation

Ok, so how should you approach test automation to avoid the bloated suite?

Don’t assume new functionality needs new E2E Tests, consider all type of automation!
  1. Update existing tests to cover new functionality.
  2. Remove any now-obsolete or redundant tests, or combine tests.
  3. Test the general case at a high level, then move permutations of that test into smaller, lower types of test.
  4. Add new, high level E2E tests only if absolutely necessary.
  5. Introduce a new type of automated test if this functionality cannot be covered by any existing type of test.
  6. Modify the system architecture to enable a new type of automated test.
  7. Continually and critically evaluate the health of the full suite of all test types with the full development team.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Blake Norrish

Quality Engineer, Software Developer, Consultant, Pessimist — Currently Sr Director of Quality Engineering at Slalom Build.