Approach to ensure quality as an absolute

With Best Testing Practices, Test Observability and a Cutting-edge Test Automation Framework

Abstract

Testing is an essential aspect of software development, especially in the context of continuous integration and deployment. As software becomes more complex, it becomes more difficult to ensure its quality. We want to be able to rely on anticipating and monitoring disruptive incidents and challenges that impact our delivery.

In this article, we discuss the importance of testing frameworks in ensuring software quality and accelerating software delivery to positive impact time-to-market. We examine various testing practices, including unit testing with mutation and static code analysis, isolation testing, contract testing, API testing, and end-to-end testing. We also examine test observability and its role in supporting test automation.

Introduction

Nowadays, test automation frameworks have become an integral part of delivering high-quality software. These frameworks are more than just tools that execute test cases and scripts. They provide a holistic understanding of the system, defined coverage strategies and quality gates to ensure the sustainability and reliability of the software over time.

In addition, test automation frameworks provide advanced visualization, reporting and monitoring capabilities that enable teams to improve software quality while reducing manual effort and time spent on testing. By standardizing testing approaches, these frameworks also help teams meet end-user expectations and positively impact time-to-market.

Due to these essential features, test automation frameworks can be considered as a system of their own that plays a crucial role in the software development lifecycle.

Technical Mindset

To ensure the delivery of high-quality software products, it is important to adopt a technical mindset when it comes to testing frameworks.

This mindset involves focusing on three key aspects:

By adopting this technical mindset, teams can ensure that their testing process is comprehensive, efficient, and integrated into the development process from start to finish.

Testing at the Right Level

There are different types of tests that can be integrated into a test automation framework. These include unit tests, isolation tests, contract tests, end-to-end tests, and API tests. Each type of test serves a different purpose and can be automated to some degree.

In addition to these tests, there are several other critical types of tests that should be included in a testing strategy. Smoke tests (quick and simple tests that verify basic functionality), regression tests (to ensure that changes or additions don’t affect existing functionality), security tests (to identify potential vulnerabilities and mitigate security risks), performance tests (to evaluate the speed and efficiency of an application), and accessibility tests (to evaluate the accessibility of an application for users with disabilities) are all important to ensure that an application or system functions correctly and is free from defects or vulnerabilities.

As a result, this provides a high-level overview of the tests mentioned as part of the Testing Pyramid with the main focus on Test Automation.

1. Unit testing with mutation and static code analysis

Unit testing with mutation testing and static code analysis are important components of a comprehensive testing framework, as they help identify bugs and vulnerabilities in the code early in the development process.

Unit testing involves testing individual units or components of code to ensure that they work as expected. This type of testing is usually performed by developers and involves writing test cases for individual methods or functions of the code. Unit tests can be automated and run frequently during the development process to detect bugs early.

But mutation testing is used to measure the quality of unit tests. In this technique, small changes (mutations) are made to the code and then the unit tests are run to see if they detect the change. If the unit tests do not detect the change, it may indicate that the tests are not thorough enough or that they are not testing the right things. Mutation testing can help developers improve the quality of their unit tests by identifying areas that need more attention.

In addition, static code analysis involves analyzing code without actually executing it. In this technique, special tools are used to analyze the code for potential errors or vulnerabilities. Static code analysis can help developers identify issues such as coding errors, security vulnerabilities, or performance problems before the code is actually executed.

So when we run both unit tests and mutation tests, we ensure that each unit of functionality is properly covered, but how do we know if that coverage is sufficient? This is where quality gates come in to enforce certain quality standards before code is allowed to move through the development pipeline. Quality gates can be used to ensure that code meets certain criteria before it is allowed to be deployed. This verifies overall quality development standards, coverage, and more.

2. Isolation testing

Isolation testing, also known as component testing, tests individual software components in isolation from other components and without dependencies on other parts of the system. The goal of isolation testing is to identify problems in the individual components of the system rather than in the interactions between them.

Isolation testing is particularly useful for identifying problems in complex systems where it can be difficult to determine the root cause of a problem. By isolating individual components, developers can identify the source of the problem and address it more effectively.

While unit tests can also be considered a type of isolation testing. However, given their primary purpose, the following techniques are known to support isolation testing as well:

• Mocking: In this type of isolation testing, certain parts of the system are replaced with simulated components, or “mocks” This allows developers to test the behavior of a particular component in isolation, without having to rely on other components that may not yet be available or fully functional. For example, you can create a mock that simulates the behavior of real functionality and use it to test the integration of your application.
• Stubs: This is similar to mocking, but involves creating a simplified version of a component to test its behavior in isolation. For example, you can create a stub function that simulates the behavior that you want to test.
• Integration testing with fake dependencies: In this type of isolation testing, the dependencies of a component are replaced with fake or simplified versions of those dependencies to isolate the behavior of the component being tested.

3. Contract testing

Contract testing is a form of software testing that aims to verify the interactions between different microservices or components in a distributed system by focusing on the contract or interface between them. The goal is to identify and fix integration problems early in the development process before they become more difficult and costly to fix. To achieve this, developers can use automated contract testing tools such as Pactflow and Spring Cloud Contract to define the contract between components and generate tests to ensure compliance with that contract, using both consumer-driven and provider-driven approaches.

4. API Testing

API testing verifies that an API meets functionality, reliability, performance, and security requirements. Functionality tests ensure that the API works as expected, performance tests measure the API’s ability to handle the expected load without crashing or slowing down, and security tests ensure that the API is secure and protected against unauthorized access and attacks.

Compared to contract testing, API testing focuses more on the functionality, performance, and security of the API. While contract testing verifies interactions between the client and the API and ensures that changes to the API do not affect the client’s functionality, API testing focuses on ensuring that the API works as expected, regardless of the client. In addition, API testing can help identify and fix performance and security issues before they impact the client.

Functional tests can be automated, and API testing tools such as Postman, Swagger, and SoapUI can be used to create and execute test cases, also using custom test automation solutions based on restassured or Spring Openfeign or Retrofit or any other HTTP library. Performance testing can be performed manually or automated, and tools such as JMeter or Gatling can be used to simulate various load conditions. Security testing can be performed using various tools such as Burp Suite or OWASP ZAP.

While contract testing and API testing complement each other, API testing is critical to ensure that the API works correctly and meets performance and security requirements.

5. End-to-end testing

End-to-end testing, at UI level, is a type of functional testing that verifies the behavior and operation of the entire system from start to finish by testing the various components as a single unit, ensuring that all components work together as intended and identifying any problems that may occur when different modules interact.

End-to-end testing can be done manually but is often automated to save time and reduce the risk of errors, automating repetitive and critical scenarios. Automated end-to-end testing involves creating scripts that simulate a user’s journey through the system, including entering data, performing actions, and navigating between pages and components. Tools such as Playwright, Selenium WebDriver, TestCafé, or WebDriverIO can be used to interact with web pages and perform various actions that mimic the user’s interaction with the web page.

Automating end-to-end front-end testing can be a good idea if you are testing scenarios that are time-consuming or repetitive, or if the user journey involves multiple interactions with different parts of the application. However, it may not be the best option for all scenarios, especially if you are testing rapidly changing or unstable systems, or if the test environment is not well-defined or consistent. In these cases, manual testing or a combination of manual and automated testing may be more appropriate. You should also consider the cost and effort associated with creating and maintaining automated end-to-end tests, as they can be complex and require a significant investment of time and resources.

What are the factors to consider when determining the percentage of testing for a software project, and how should testing be divided among different types?

To provide a good answer, it is important to prioritize the testing effort based on the criticality of the various components and functions. For example, if the system has a complex API, more importance may need to be placed on API testing to ensure that it functions correctly and meets project requirements. Similarly, if there are critical dependencies between components. In this case, you should focus more on isolation testing to ensure that these dependencies work correctly.

While there is no universal approach to determine the percentage split of testing, a general guideline is that most of the testing effort should be devoted to unit tests, as this is where most bugs are usually found. The remaining tests can be distributed among other test types such as isolation, contract, API, and end-to-end tests, depending on the specific requirements and goals of the project. A common example of the distribution could be:

• Unit testing: 50%
• Isolation testing: 10%
• Contract testing: 10%
• API testing: 20%
• End-to-end testing: 10%

It’s important to note that this distribution is just one possibility and may need to be adjusted based on the specific requirements and goals of the project. Additionally, the distribution may also vary depending on the available resources and time for testing, and also how mature the system is, the project, and the teams. Matury levels (Initial, Managed, Optimized, etc.) per team are also important to understand how to distribute the testing coverage and plan accordingly.

Test Observability

Test observability refers to the ability to collect, analyze, and visualize relevant information about the status and outcome of test execution and its tests. This includes reporting, a data representation of test execution information, integration with test management tools, and real-time dashboards for monitoring.

Data representation of test execution information as key data includes aggregation and analysis of data generated during the testing process. This can include metrics such as test details (name, category, environment, priority, etc.), test coverage, test execution time, and defect rates that allow you to track the progress and effectiveness of testing.

Reporting involves creating test results that can be shared with stakeholders automatically and directly, including developers, testers, project managers and other team members. These reports should include details about the tests performed, their results, and any issues encountered, and also evidence such as screenshots, recordings, API calls, etc.

Integration with test management tools allows teams to connect their testing efforts with other software development processes, such as issue tracking, project management, and continuous integration/continuous delivery (CI/CD). This can help streamline workflows and improve overall efficiency.

Real-time dashboards provide visual representations of test execution information, allowing teams to quickly identify trends, issues, and potential areas for improvement. This can help to monitor test automation status and evolution or degradation and facilitate collaboration and communication among team members and can help ensure that everyone is working toward the same goals.

Conclusion

Implementing a modern testing framework that is consistent with continuous integration and delivery is critical to ensuring software quality and reliability. Such a framework provides faster feedback, improves quality, increases efficiency, reduces risk, and fosters collaboration between developers and stakeholders.

Shift-left testing, as well as various types of testing such as unit testing with mutation testing, static code analysis, isolation testing, contract testing, end-to-end testing, and API testing, are essential to detect defects early and reduce the cost of fixing them. The integration of AI and machine learning can further improve the testing process.

However, implementing a test automation framework is not a one-time solution. Ongoing monitoring, improvement, maintenance, and test observability are necessary to ensure software effectiveness and continuous improvement.

Looking to the future, advancements in AI, cloud-based testing, and virtualization technologies hold great promise for testing automation frameworks. With their integration, frameworks will become even more efficient and reliable, leading to faster software delivery and improved customer satisfaction.

In conclusion, testing automation frameworks are an essential tool for modern software development. By continuously improving and optimizing their frameworks, developers and organizations can enhance their software products and remain competitive in an ever-changing industry.