A pied piper named “UI test automation”
There was once a pied piper named “UI Test Automation” (also fondly & interchangeably called as automation, automation testing, test automation) who promised to help speed up software testing by giving tools & ways to automate functional tests that interact with software-under-test just like users would — via the Graphical User Interface. Stakeholders in the software development process, especially software testers, were amazed by this magic ✨and believed this is the panacea that would cure all their testing woes.
With the help of the pied piper, testers, in hordes, started converting the bulk of their “manual” test-cases into automated end-end UI tests. More and more testers from different parts of the software world started following the flock. Flocks became crowds and crowds became even larger crowds. It was not too long until almost every tester became obsessed with this approach of test automation. UI test automation as an approach & it’s tools became hot skills in resumes. The software industry soon adopted a new way to tell testers that they should continue to wear blinders and stick to traditional means of black-box style, end-end testing. This time, by expecting them to convert test-cases into UI test automaton “scripts”. Easy peasy!
Fast forward to 2022, and the pied piper is still a celebrity on the pedestal that we testers, blindly follow.
Why is that the case? And does it matter?
The rise of UI test automation and the fall of a meaningful, context-driven test automation strategy
“When there is demand, there is supply. This is a market driven world”. This is the most convenient way to justify a lot of weird things that happen in the world. With the increasing demand of UI test automation as a skill, we have been witnessing below patterns.
Dedicated teams to write end-end UI tests in silos
A test strategy focused purely around automated end-end UI tests brought along the promise that tests that simulate user behaviour would be a perfect way to achieve testing with speed. Stakeholders in software development often consider UI test automation as a highly trivial & simple activity of writing “scripts” that push buttons, enter text, select checkboxes, validate presence/absence of components on the screen.
Trivial enough to delegate it to an isolated, dedicated or external team (read outsourced) that converts thousands of manual test-cases to “scripts” but may not have the product knowledge, application architecture context, visibility of business priority & user impact to write effective tests. Such teams would just do what they are told to do.
This has given rise to a lot of testers who are running behind becoming UI automators instead of becoming test automation specialists.
Test automation specialists are testers who treat test automation as a tool agnostic, interface agnostic craft. A craft that is driven by information, context, strategy & value, just like testing.
Challenges vs actual value from end-end UI tests
Automated tests of any kind provide value only if they are implemented properly. Given the nature of end-end UI tests — there are a TON of problems that need to be solved in order to derive value from them. Many of them are within our control, many outside.
Below are the typical problems an organisation goes through when their test automation strategy is entirely focused on end-end UI tests:
- Large number of tests turning out to be consistently flaky.
- A single instance of execution of these tests taking too long to finish.
- Testers taking too long to implement these tests.
- Teams losing confidence in these tests and releasing features without looking into the test results.
- Bugs leaking to production.
These problems usually have below causes :
- Trying to automate all the “test-cases” as UI tests, thus spending way too much time on the test automation task instead of doing what testers do best — i.e., learning/exploring the product, gathering & analysing information, highlighting gaps & collaborating with product teams. Funnily, nowadays it is okay for testers to not be good at testing but great at UI test automation.
- Treating test automation as only the tester’s problem and not the team’s problem. Ownership of test automation is purely left to testers while it’s okay for developers to not contribute to any kind of automated tests. This often leads to a culture where there is an over-reliance on end-end tests and testers spending too much time on them.
- Poor tester:dev ratio in the organisation is also often a cause to the above problems. If you have 15 developers in your team and just 1 tester, it’s a recipe for disaster if just the tester is contributing to test automation.
- Poorly implemented test code, frameworks and execution infrastructure with no dedicated folks to fix issues, maintain and improve them (lack of a focused COE / test platform team)
- Testers lacking knowledge and experience of testability and testing across the system architecture.
Implementation of effective end-end UI tests IS NO JOKE. It is as good as production code development. These tests are time consuming to implement, slow to execute, prone to flakiness, require a lot of thinking and planning in order to get them right. Such tests also tend to become a maintenance and collaboration nightmare IF they are not done right. Given the time, effort & investment required to build and maintain these tests — does it make sense to have UI test automation as the only test automation approach?
Evolved software architectures, evolved development methodologies, un-evolved test automation approaches
Modern software development tools & architectures have evolved a lot. Look at any industry that is building software and you will see there is adoption of modular & loosely-coupled systems. Modular and loosely-coupled systems allow better testability.
Of course, loosely-coupled systems alone don’t ensure complete testability but they do allow us to test a system without interacting with the UI. At times there has to be an intentional effort to make a system testable.
With Agile (or iteration based) software development methodologies becoming a norm everywhere, teams are becoming more collaborative, dynamic & cross-functional. The practice of building and testing a feature in small chunks has also made it possible for testers to be involved early in the lifecycle of the feature’s development. This also means that automated tests can be built early, which also means that testers don’t need to automate all tests at the UI layer.
But in reality, many testers still haven’t evolved to think of test automation as an interface agnostic activity where their contributions to test automation can depend on the interface that is being built in the sprint. They will typically wait for the entire feature to be released and then pick up the task of converting their documented test-cases to UI tests usually in the next few sprints. With this approach, test automation becomes less important, less visible and less valuable.
Is it the pied piper’s fault though?
Absolutely not. End-end UI test automation is necessary to gain confidence that our systems work as a whole for the user when they interact with the UI. At times we might even need to focus more on these tests if the context or nature of the application demands it. But are we doing it the right way? Are we thinking of test automation strategies holistically? Are we looking at all the directions? Are we asking the right questions?
What can we do about it?
As testers
As testers, I think we are often too happy with our comfort zones. We treat UI test automation as an absolute truth. We are comfortable being told what & how to do something by someone else, instead of thinking for ourselves what a better approach to solve a given problem could be, based on context. This can change by changing how we learn test automation, how we collaborate with developers and how we practice test automation at our workplaces.
On learning test automation
Firstly let us start understanding and discovering test automation as a craft holistically instead of treating it as a way to get that salary raise or a promotion. Let’s stop thinking that test automation = something that is ONLY done via the UI or only done for end-end flows where the entire system/environment is available to you. Does the term “test automation” contain the word “UI” anywhere? No, right? So why do we need to strictly restrict ourselves only to the UI when it comes to learning or implementation?
Typical test automation spectrum:
- Unit tests
- frontend unit tests — Unit tests written to verify business logic in the frontend code.
- backend unit tests — Unit tests written to verify business logic in the backend service code.
2. Component tests
- frontend component tests — Tests larger than frontend unit tests written typically to verify UI component behaviour independently in isolation without real dependencies.
- backend component tests — Tests larger than backend unit tests written typically to verify a backend service works correctly in isolation without real external dependencies.
3. Integration tests
- frontend integration tests — Tests that verify integration of multiple components together in the frontend code.
- backend integration tests — Tests that verify integration of multiple components (either inside a service or between multiple services) together in the application backend.
4. End-end tests
- End-end UI tests — Tests that verify a complete user journey via the GUI and the entire environment is setup & running.
- End-end API tests — Tests that verify a complete user journey via the API (usually the apis exposed to the frontend) and the entire environment is setup & running.
As you can see above, test automation = automation of a test/check against any interface. Not necessarily a GUI. Test automation is a whole wide spectrum that includes unit, integration and end-end tests.
So in my opinion, when anyone (& especially a tester) starts their test automation learning journey, they should start from learning atleast the basics of unit tests and then move to higher levels of tests (component/integration/end-end).
“This is ridiculous and shocking 😱. How do you expect testers to learn unit testing? That’s what developers do. Are you crazy😤?” is what many testers might think.
But it’s not ridiculous. When you are learning test automation, the first thing you usually do is learn a programming language right? Do you just learn the syntax and rules of the programming language? No. You also write programs to implement some algorithms, or applications as part of your practice. So why not learn how to write unit tests to test the practice applications/algorithms?
Doing so gives you an idea of what an automated test looks like in the simplest form with no thousand bells and whistles that you have to learn for writing end-end UI tests.
Meanwhile, understand the concepts of code coverage as well while doing so and you get a really good idea of how developers really write unit tests and it also gives you an edge where you can easily communicate with developers on a meaningful test automation coverage.
Does that mean you should be writing unit tests for the SUT? No. I think that’s best left for the developers. But you will at least know how, why, when, where they are or are not writing unit tests.
This will also allow you to define a better test automation strategy for feature development in your team where you can choose to decide what kind of tests you or your team should focus on based on the context.
On strategising and implementing test automation
The act of strategising and planning for test automation is way more important than that of implementing the actual tests. In order to come up with a strategy, as testers we should start asking ourselves and our team questions.
What kind of questions?
- What does the technical solution for product requirements look like?
- What are the frontend and backend changes being done?
- What are the most critical or impactful requirements? (~ what requirements should have automated tests without compromise)
- Where does most of the business logic fulfilling the requirements go in the system? Is it in the backend, front-end or both?
- How & why should a specific scenario/requirement be automated as a specific type of automated test (Unit/component/integration/end-end)?
- Where should the automated tests run (environments)?
- When should the tests be automated?
- Who should be automating the tests?
- Which tests should be automated first and why? (unit/integration/end-end)?
- How testable are the features that we want to write automated tests for? Are they easily testable programmatically at the level/type we have chosen? Or do they need complex setup? Are all the variables easily controllable from the tests?
The output of these questions and discussions should then become an input to your actual test automation effort.
If after answering these questions, you feel that end-end UI tests are the only right way to verify the correctness of the feature, then so be it.
As developers
Developers in a product development team should be equally invested (even if not fully participating) in test automation of any type. They should work together with tester(s) in the team to derive a strategy based on above questions for programmatically checking the correctness of a feature and pick up test automation as per the agreed strategy and plan. You can especially help testers by writing UI unit/component tests or backend unit/component tests for requirements that testers thought of covering with end-end UI tests.
If testers are responsible for end-end UI test automation, developers should support the testers in ensuring the features are testable in an end-end manner and if not, enable testability for them.
If you do not have clarity on what kind of end-end tests are being written by the testers, ask them to get you involved in the execution results.
Automated tests are most useful when they are written and executed while the feature is in development.
As leaders or stakeholders in testing
As leaders or stakeholders in testing, we often underestimate the huge amount of time and effort it requires to build and maintain end-end UI tests. Let alone the frameworks and execution infrastructure. We often ignore considering testability as a first class citizen so that tests can be automated where it makes the most sense, and by the most sensible person. We often ignore the fact that there are a lot more valuable things that testers can also spend time on apart from just test automation.
Leaders are the best partners of testers in spreading and advocating for a culture of meaningful automated tests that the entire team feels responsible for. Apart from testers, they too should participate in discussions around test automation strategies, test automation coverage to understand the state and quality of test automation in the teams they lead.
As a UI test automation tool/platform vendor/creator
Since the last decade we have witnessed lot of new amazing platforms and tools that are trying their best to make end-end UI testing easy, fast and scalable.While that is great, it’s equally important for these platforms to be transparent and vocal about test automation strategies so that organisations build meaningful automated tests to derive the most value, instead of focusing entirely on UI. For example, check how Selenium takes a stance on UI automation.
As a testing community
As a testing community we have been ruminating a lot (and I mean A LOT) on the same UI test automation topics over and over again. Let’s obsess less over UI automation tools and start talking more about context, strategy and value-driven test automation.
Here are some further reading/reference materials from some amazing folks of the testing community, as food for thought:
- Just say no to more end-end tests by Mike Wacker
- Say no to (more) Selenium tests by Bhumika S, Anand Bagmar
- Unit tests & code reviews: the bedrock of software quality by Heemeng Foo
- A discussion on automation across layers by Lavanya Mohan & Ajay Balamurugadas
- 100,000 e2e selenium tests? Sounds like a nightmare! by Alister Scott
- The practical test pyramid by Ham Vocke
- An insightful post on UI test automation by Bas Dijkstra
- Pyramids are ancient- Test Automation Strategy by Richard Bradshaw
- Redefining test automation by Richard Bradshaw
- Fixing a Test Hourglass by Alan Myrvold
- A comment worth reading in the Fixing a Test Hourglass blog
- Say TaTTa to Your TuTTu: Avoid Anti-Patterns and Target Your Testing and Checking by Mark Winteringham
- Devs should love your tests by Carlos Kidman
- An insightful Twitter thread by Gaurav Singh on test automation
