What makes Continuous Testing possible?
Co-author: Dragos Tanta
As Wikipedia describes it, Continuous testing “is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate”. But that is only a high level description. When you deep dive into the implementation of continuous testing, you`ll face the demons of distrust and take the hard and bumpy road of convincing others.
Continuous testing will take the same road as all the other processes that took people out of their comfort zone; like Agile when it first appeared. At first it sounded impossible to be implemented, a fantasy that will not work in real life (at least not when we speak about a system of systems). After several years in the market, Agile is a strong mindset, that everyone follows and now agrees on its usefulness. If it is something that we should learn from the other success stories, is the engine that eventually makes them possible.
So, what would make Continuous Testing possible in your organization?
1. Continuous Integration
2. Continuous Delivery pipeline
3. Automated tests
But is this enough?
We say: NO. You still need the correct mindset to accept and implement all the above.
Even with the right tools and knowledge and with the experiences of various companies that already tried and implemented it, it’s not enough.
Not defining the process, not defining the tools, but making people involved and make them seeing the benefits while you take them out of their comfort bubble is the key success factor.
Continuous testing came with benefits on software product but with huge impact on the software team. Especially on the QA team.
One thing is clear: We cannot have an efficient continuous delivery or continuous deployment process without continuous testing. Is like a car with no wheels, it`s still a car but It just doesn`t move around. And we cannot avoid the continuous delivery either. The world has tasted the fast release cycles and wants more. And as we cannot have continuous deployment process without continuous testing, we cannot have continuous testing with traditional testing techniques and mindsets. Long and heavy manual testing processes, huge frameworks with automated GUI tests that try to test “exhaustively” all the business logic will no longer have a place.
For UI testing, Selenium is a great tool, that is widely used and has a great support on the online community. Nowadays we can find frameworks that come as a wrapper on top of Selenium with already built in features like: browser drivers management, cloud integration, assertion, reporting etc. There is no need to invest in writing these features from zero. With Node.js based frameworks like Nightwatch.js or Webdriver.io you can setup and start creating a functional test script in just minutes. They support integration with external libraries for logging, reporting or everything else needed from automation perspective.
For micro service tests, java frameworks like Rest-Assured, allows a fast and easy setup for verification of requests and responses of Rest API endpoints. For Node.js, frameworks like Mocha can be used with Chai on top as assertion library for exactly the same purpose. All these open source out of the box frameworks, help us to be up and running quite fast, and focus directly on the testing scripts, rather than investing time to build all these functionalities from zero in the framework. We must take advantage of the open source tools provided by the online community.
The only option for QA professionals (except a career change) is to ride the wave. Testing is not just about finding failure in the code, testing as a discipline is also about assuring the best possible experience for the end user. Although some developers see the tester as Devil`s Advocate the tester is actually the User`s Advocate.
Many QA professionals seem that they have forgot their role and are just stuck in rigid processes and procedures each one with a fancier name than the previous one, becoming bottlenecks in the deployment process.
Maybe this is why Google preferred to change the tester’s title and role.
@Google SETs (Software Engineers in Test) and TEs (Test Engineers) exists in a separate and horizontal focus area called Engineering Productivity. They are “on loan” to the product teams and are free to raise quality concerns and ask questions about functional areas that are missing tests or that exhibit unacceptable bug rates. SETs spent close to 100% of their time writing code, but they do so in service of quality rather than coding features a customer might use1.
The result? According to The State of Continuous Integration Testing at Google by John Micco from ICST2017, Google has 4.2 million individual tests running continuously and 150 million test executions / day (averaging 35 runs / test / day). Impressive I would say, even if we are talking about Google.
In continuous testing we need to be at least as fast and dynamic as the development process. And to do that we need to have people with the passion for building great products. Eventually is the passion that drives the mindset towards success, not the requirements. What it means is that the testers have lost the comfort bubble of their roadmap, with time allocated for the preparation of tests, time allocated for test execution and so on.
In continuous testing everything happens NOW, because once the deploy is done, we need the feedback NOW if the build is stable enough to go to next stage, which can mean even production. So we need to test good, fast, automatically and continuously. Test automation must happen at least in parallel with development, if not possible sooner. We need to build only the relevant automated tests and we need to do it fast. We also need to automate our automation. It worth nothing to have automated tests if they are not automatically triggered by the build pipeline to provide the feedback when and where we need the most; i.e. right after new code deployment. And if we need to give the feedback fast, we need to run in parallel and scale as fast as a developer spells “commit”. And for this to be achieved, we need people with a build-in quality mindset.
With tools like Selenium Grid, that allows running multiple tests on different browsers in parallel, and virtualized environments based on containers ,that allows easy setup and scalability, the chase for rapid automation remains only a matter of mindset.
We can automatically deploy Docker containers in matter of seconds to run the testing scripts, thus being able to provide feedback as soon as needed for each stage of the deployment process.
Can we guess the future of continuous testing?
For sure continuous testing will stop becoming an ideal to be reach by companies but rather an normal day to day process, and the traditional software testing techniques will be just a memory. The use of continuous testing at large scales will open new and new opportunities for improvements.
Processes like: predictable test runs based on advanced statistical algorithms, automatic scripting based on usage statistics will be the next big challenges for QA professionals.
And from that point, the race for improvements will definitely be completed by the usage of artificial intelligence in the testing process. The presence of artificial intelligence in development and testing processes will soon not be just a buzzword, but a day to day reality, and soon enough test professionals will stop chasing the benefits of open source tools and start chasing the AI testing tools.
Happy testing!
What’s your opinion? Please comment below
1 Source: Book “How Google tests software” by James A. Whittaker, Jason Arbon, Jeff Carollo
