The Ecosystem Of Test Automation: Environment
On Adventure Through Shapeshifting Test Automation Strategies
Being a test automation engineer is like being an adventurer. Imagine yourself exploring the surroundings of your test automation garden, discovering the ecosystem of test automation.
Sometimes you hike in the humid jungle of Java, or you might discover a Python! Another time you might be on a pebble beach of microservices, or is a large monolith blocking your sunlight?
Different environments have in different ways impact on test automation. In order to achieve a sustainable test automation garden for your environment, it is very important to find the best suitable strategy and implementation of that strategy. In this blog I will take you on adventure through a diversity of application landscapes, shapeshifting test automation strategies and the tools used. Check out some high-level examples, get inspired and, release the adventurer in you!
My adventure began with the assignment to coach ten teams with test automation. Ten teams within a complex application landscape that touches a large part of the technology spectrum; from open source languages and frameworks like Angular, .NET, Java, SpringBoot to package software like Oracle SOA Suite and Siebel. Not quite a walk in the park, so I decided to go on a ‘Walk in the park’ to align on the strategy of test automation and determine the implementation of that strategy. I brought my garden toolkit and pulled out my favorite strategy; the Test Automation Pyramid.
This model visualizes the ideal distribution of automated tests, based on the properties of these tests. The higher in the pyramid, the more dependencies and greater costs of maintaining tests, but also the greater the relevance of the tests. During the ‘Walk in the park’ it appears that the current distribution of the automated tests is more shaped like an ice cream cone. The opposite of the Test Automation Pyramid and a well-known anti-pattern.
The majority of the automated end to end tests take place in the GUI. A lot of flaky tests with a lot of dependencies verifying huge flows, resulting in a lot of maintenance and frustrated people. Fortunately, this anti-pattern can be combated by implementing the Test Automation Pyramid. For the teams a clear concept to grasp, but quite a challenge to apply in practice.
Discovering the real Test Automation Pyramid
The questioning starts with the definition of a unit test. A unit test verifies the specific behavior of a unit; the smallest testable part of the application. For Java or .NET applications the smallest testable unit is a single function/method within the code.
But what is a unit in Oracle SOA Suite, which is used to assemble multiple components into one SOA application? Testing the code means testing the software suite itself, which is definitely not the intention. So, in this case the smallest testable part is rather a component, than a small piece of code.
The definition of an integration test also depends on the environment and perspective. What is meant by integration? Integration within an application or across multiple applications? Where does one team end and another team start? Talking about start and end… what is ‘end to end’?
It was clear that the teams needed something more tangible than a model. Visualizing the scope of the different tests really helped. For the team working with Oracle SOA Suite, together with a developer I created the high-level example below.
It is a simplified view of a so-called BPEL component, where a business process is executed. During the process data is accessed from services/databases via other components. With a unit test these external components are simulated, and in an integration test actual services are used. Sharing real life examples of a unit and integration test within the specific context would be the next step.
For comprehensive and extensive examples of tests in a simple microservice architecture with Spring service and database integration, check out the ‘Practical Test Automation Pyramid’.
Add more layers to the pyramid
The scope of the test may differ per context, but also the number of test layers. I currently work in an environment where a layer between unit and integration testing has been identified: component testing. The focus of a component test is on functionality and business rules. A frontend or backend application is considered a black box, and all communication with the outside world is simulated through mocks and stubs. The connection and integration with other applications is tested with an integration test.
Rest Assured and Spring Cloud Contract are used to implement the backend component and integration tests. In the frontend applications, those tests are implemented with GUI test automation tool TestCafé. The end to end tests are application-transcending user flows and will also be automated via the GUI.
There are situations that the Test Automation Pyramid takes on other forms.
Test Automation Honeycomb
Adventure is guaranteed in the microservice world of Spotify. Their software scales to millions of users on multiple platforms. That requires a robust test automation strategy. Spotify based their strategy on the Test Automation Pyramid but experienced that having too many unit tests did more harm than good. It resulted in too much maintenance and slower iterations.
And it didn’t outweigh the added value. In their context integration with other services is the greatest complexity of a microservice, not the code of the service itself. The focus shifted to integration test, turning the pyramid into a honeycomb. Check out their story here.
Test Automation Trophy
There is clearly a winning mentality with the Test Automation Trophy introduced by Kent C. Dodds. This model is based on efficiently building confidence in the code with the available tooling in frontend applications.
Static code analysis tooling like ESLint or TypeScript enforce certain rules and code conventions, which already builds a lot of confidence in the code. Jest is the suggested JavaScript testing framework for both unit and integration testing, with the advice to focus more on integration testing. Kent C. Dodds explains that with integration testing there is a great balance on the tradeoff between confidence and speed/expense.
Being on an adventure through different application landscapes provides insight into different perspectives on test automation. Although it is a Test Automation Pyramid, it is not set in stone! It serves as a great foundation and starting point to determine the best suitable strategy and implementation. All various shapes and forms might be overwhelming. But if you make sure you keep asking the important questions, you will be alright.
- What functionality is most important for the business?
- Where lies the greatest complexity of the application/architecture?
- What is the tradeoff between confidence and speed/expense with the available tooling?
Based on the answers, focus more/less on the various test layers. The result might be a pyramid, honeycomb, trophy or backpack for your adventure. Just be flexible and go with the flow!
🌱 Want to read more about the Ecosystem of Test Automation?
Check out my next blog; examine the role of the gardener, how it is influenced by the ecosystem, and how other areas of expertise can help to maintain test automation and ensure quality in a fast-changing world!
