Release High Quality Products: Breaking down and reassembling automation to gain its full potential

Alternative System Validation and Test Automation Approach

Intro

The tech industry is currently undergoing significant transformations, marked by the 22–23 high-tech crisis and a growing demand for infrastructure roles such as automation and DevOps. My new position opened the door for me and allowed me to build this field from scratch, which led me to extensively explore and map the realms of automation, DevOps, validation, and testing infrastructure engineering. Through this process, I gained insights into the responsibilities of these teams and the challenges they encounter.

After more than decade of climbing the career ladder both vertically and horizontally within teams and organizations, I’ve come to understand 2 things:

• Thes roles differ between organizations, and I know nothing, I mean I know something but it is a real small drop in an ocean.
• There is high significance of mapping out the roles, challenges, and values of such teams for myself and the success of the organization I am part of.

Let’s map the challenges

Product health maintenance challenges

1. Difficulty in maintaining sync between all product components:
   This challenge arises due to the complex nature of modern software systems, which may comprise multiple interconnected components, modules, or services that work together to deliver the intended functionality. When changes are made to one component, it can impact the behavior of other components, leading to potential integration issues and unintended consequences.
2. Maintain automation infrastructure in a fast delivering echo-system:
   automation software needs to constantly adapt and align with the changes in the product. This requires a dynamic and adaptable approach to automation and continuous integration and delivery processes. If the automation project is not an integrated part of the entire product project, it may become difficult to maintain stability and alignment between the automation and the product, leading to delays and issues in the system validation process
3. Collaboration and communication:
   Effective collaboration and communication among different teams involved in system validation and CI/CD processes can be challenging, particularly in large organizations or complex systems. This can include issues such as coordinating testing efforts, sharing results and findings, and ensuring that all stakeholders are informed of the system’s current state.
4. Timely releases and reliable products without last-minute surprises
   The constant pressure to meet deadlines often leads to last-minute surprises, introducing unexpected issues that can jeopardize the product’s dependability. Achieving a harmonious blend of timely quality releases requires efficient workflows, and proactive technical problem-solving tools.

Automation and Debug systems challenges:

1. Supporting testing across different platforms ,devices and environments:
   This challenge arises due to the need to support testing across different platforms, devices, tools, and environments. In addition to the increasing complexity and diversity of modern software systems, which may need to be tested on various configurations to ensure their compatibility, performance, and reliability.
2. Resource management:
   Running a large-scale test automation suite can require significant computational resources, which can be expensive and difficult to manage. This can include issues such as scaling the test environment, ensuring adequate storage and memory, and optimizing test execution times.
3. Efficiency and accuracy of defect detection:
   In most cases, the Automation systems we are discussing are designed to detect defects in complex software products. However, when a defect is identified, it can be challenging to determine the specific component that is causing the issue. This is due to the fact that numerous components are integrated during each automation execution, making it difficult to pinpoint the source of the problem.
4. Inability to measure test effectiveness in complex systems:
   Typically, the Automation systems we are addressing are designed to carry out a large volume of tests, system characterizations, and real-time metrics. However, it can be challenging to determine which tests are the most productive and which ones are not. Without this information, we may waste time and resources on ineffective processes.

Reliable test plans challenges:

1. Managing test data for large and complex systems:
   The quality of test data is critical to the success of the testing process. Poor quality data can lead to inaccurate test results, which can in turn lead to bugs and other issues in the system. Managing qualified data is getting harder as the system changes over time, test data may need to be updated or modified to reflect those changes. This can be a complex and time-consuming process, particularly for large and complex systems.
2. Prioritizing test cases based on various criteria:
   This challenge involves selecting a subset of test cases that can adequately represent the entire test suite while also providing maximum coverage of the system under test. This task becomes more complex when there are various criteria in a fast changing environment that need to be considered.

Combine the challenges with couple of values

Be a real combat support

Ease on product development lifecycle and deployment processes DO NOT make it harder

Be a good parent for your babies

Taking full ownership and management of infrastructure and tools developed for the RnD team

No discount’s in here

Breathe and live product’s quality and health during the whole lifecycle, debug and understand each issue either in your framework or in the product.

It is all about the product

Remember: you develop SW product that is representing a big complicated system. NOT bunch of i-related test scripts

Scalability is not just a buzz word

The processes you develop right now that should support X amount of components and tests may become 10X in a time, make sure your system is build adoptively

And finally define the roles

Technical Roles:

1. CICD Tools and pipelines development
2. Automation Infrastructure development
3. System tests development
4. Automated tools for the organizational purposes
5. Automated system debug tools development

Methodological Roles:

1. Release management
2. SW Lifecycle methodologies implementation
3. Cross team technical and personal communication

In the future posts, I would like to cover each of these challenges and explain how the definition of roles and values can help overcome them. Throughout my posts, I will try to go in between the drops and provide reliable and detailed technical information, however maintaining a macro view of the system, as each implementation, tool selection, and technology development is dependent on the organization’s limitations and needs.