Greenhouse of Test Automation: part I
The benefits of visualizing test automation
I’m proud to be Dutch! Especially since I’ve learned that The Netherlands (or: Holland) is the second largest exporter of agricultural goods in the world. A country 270 times smaller than number one; the USA. How can this be? The answer is in the greenhouses.
Why are the greenhouses of Holland so efficient? What can we learn from them to get the most out of test automation? How can visualizing automated tests bring you closer to sustainable test automation and build confidence in quality assurance?
🌱 Here starts the journey of the ‘Greenhouse of test automation’. A context-driven voyage of discovery through the world of test automation visualization using an Influx database and Grafana dashboard. In this blog series you will be provided with practice & implementation examples, and lessons learned to start your own journey.
This first blog covers the concept ‘greenhouse of test automation’, two customer cases where this is applicable and the first lesson learned.
Greenhouses of Holland
Dutch greenhouses use technology to aim for circular sustainability. The individual health of crops is measured by countless sensors. Detailed readings on soil chemistry, water content, nutrients and growth are stored in data warehouses. This data is visualized and analyzed to determine what the crops exactly need. And maybe more importantly, what the crops don’t need. In that way these impressive results have been achieved since the 2000s;
- 💦 90% reduction in water dependence
- ☢️ Almost complete elimination of chemical pesticides
- 💉 60% reduction in antibiotic use*
At the same time, the crops are kept healthy and the added value of the gardens is optimized. This reminds me of another garden, a garden of test automation!
*Source: https://www.nationalgeographic.com/magazine/2017/09/holland-agriculture-sustainable-farming/
Case 1: Garden of test automation
An automated testset of 3500 GUI tests was the team’s welcome gift when I started a new assignment in 2017. It ran daily, and every time 500 tests were failing; a true gift of challenge so to say. Surprisingly, the team paid no attention to the testset. Bugs slipped through to production, causing the test set to stop adding value.
By making test results visible on a dashboard, I confronted the team with the situation.
🌱 I called it the ‘garden of test automation’, a garden to keep green!
After six months of weeding and gardening 600 tests remained, of which one or two were failing sporadically. The team responded to failing tests, and bugs were detected by the test set again.
The dashboard played a big part in this improvement process. Pointing out the state of test automation triggered the team to do something about it. It helped determine exactly what the tests needed, and what they didn’t need. Just like with the crops in a greenhouse of Holland. In retrospect, this was my first greenhouse of test automation!
🌱 A ‘greenhouse of test automation’; where the results of automated tests are measured by test automation sensors, saved in a data storage and visualized on a dashboard. Raising awareness and motivation to maintain the health of automated tests and achieve sustainable test automation.
Case 2: an ‘Evergreen’ challenge
Together with two fellow test automation engineers from ‘deTesters’, I’m currently assigned toPort of Rotterdam. Technical and functional requirements are largely covered by automated tests, and generate quick feedback within CI/CD pipelines. Our biggest bottleneck is the time consuming process of acceptance testing performed by the business. Functional maintenance and business consultants test application transcending user flows manually via the GUI. Automating scenarios like these mean lots of dependencies, instability, maintenance and therefore a big investment. Clearly something we are reluctant to do.
Why is it that the business spend so much time acceptance testing? After all, a lot of requirements are already covered by test automation. The business is aware of the presence of test automation, but appeared to have no idea what exactly is covered. They don’t feel confident in quality assurance through test automation, which means that extensive testing is still carried out at acceptance level. So instead of automating more, we decided to make the health and added value of existing test automation more visible, and serve multiple purposes:
- Build more confidence in quality assurance through visualization of test automation.
- Identify possible gaps & doubles, to optimize the effectivity & efficiency of test automation.
This means it’s time for a new greenhouse of test automation!
Thrift shopping for tooling
What tools do we need to visualize test automation? In retrospect I’m very happy to have started the search within the context of Port of Rotterdam. Our platform team already had a setup in our application landscape for production monitoring. The data is stored in an Influx database, and visualized on a Grafana dashboard. Definitely something we could reuse!
🌱 ️ Lesson 1: Reuse available tooling
It’s cheap, already part of your context and people have knowledge about it. A platform/OPS team is definitely a good option to shop for visualization tooling!
What test automation sensors do we need to extract results from automated tests? How can you send them to an Influx database and visualize them on a Grafana dashboard? Especially from different test automation tools? Check out my second blog of the Greenhouse of test automation series and find out!
