Testing in the CD pipeline — the deficient adhesive that fractures the CI/CD architecture
Our thoughts on the testing framework, emerging trends and opportunities in the space
Software development methodologies have evolved over the decades in a direction allowing quicker development and deployment with ever-increasing application complexity. The age-old waterfall systems were replaced with agile methodologies which involved breaking down an application into smaller modules and developing each of them in parallel iteratively. Once built, the modules were integrated together and deployed as a single application either at edge or on the cloud. However, as the functionality of applications increased with large temporal variations in the incoming traffic across the different app sections, the deployment methodology also shifted to modules — and the concept of microservices exploded.
A single monolithic application is broken down into hundreds of microservices, with each one following its own development, testing, deployment, and monitoring cycles. The entire application is brought together by an underlying CI/CD (Continuous Integration/Continuous Deployment) pipeline infrastructure. The DevOps methodology is envisioned to automate all the steps following a code commit right up to deployment. As soon as a developer commits the code, the pipeline should automatically build, perform integration tests, staging tests, provision and configure infrastructure, and finally deploy the service making it live for the customers. At Pravega, we have been closely following developments in the microservices/containerization area and its implications on CI/CD. We will share our observations in a series of notes, starting with this first one on Testing in the CD pipeline.
The end goal is clear — to achieve complete autonomy across the CI/CD pipeline. However, after speaking to 20+ companies in various stages across Series-A to pre-IPO, we have observed that CD is rarely utilized to its potential. There seems to be manual interventions post the integration tests to verify end-to-end functionality, performance and conduct extensive staging tests — all in all to build enough confidence on the code to deploy in the real-world scenario. Coupled with the number of steps including provisioning of the staging infrastructure, configuring the testing environment (dependencies, databases, etc.), creation and maintenance of test cases (front-end, API, end-to-end, performance, security), running the tests, analysis and management of test outcomes, the end goal seems like a far-fetched dream. The above inferences are substantiated by extensive surveys [1–3] which point out the fact that while companies understand the significance of testing and consider the same to be an integral part of the development cycle, the extent of testing is severely limited to unit tests, followed by integration tests and minimal testing in the CD pipeline. Moreover, as pointed out in the Gitlab DevSecOps Survey 2021, testing remains as the area most likely to cause delays for the third year in a row (2019, 2020, and 2021 surveys). “Testing delays everything”, “Testing is not yet fully automated in the deployment cycle”, “Testing can be slow in both writing and running” are some of the extracts from the Gitlab DevSecOps Survey 2021. The challenges also present vast opportunities if relevant tools are built. The domain of continuous testing aims to address the challenges in the above steps.
The tools available in each of the above buckets alongside the trends observed are discussed as follows
Preparing Test Cases:
Testing in the CD pipeline is primarily done on five fronts — Front-end, API, End-to-End, Performance and Security. The figure illustrates the tools available in the market with each of them adopting different approaches to prepare test cases. The current state of testing is as follows:
- The preparation of testing scripts and the corresponding data sets is largely manual and driven by the developer/tester.
- The scope of the test cases is always under doubt resulting in additional manual interventions and delay in deployments. To save time, scenarios testing only the core functionality of the product are created resulting in avoidable bugs in production.
- Translation of updates in the testing script as and when changes are made to the microservice (front-end or API) is entirely dependent on the developer.
- There’s lack of communication and collaboration between developers and testers which can result in missing out on critical test cases as well as make the identification/pointing of errors (in a microservice/API) while testing a challenge.
The above scenario clearly results in inefficient and time-consuming testing phases, and we can observe trends in terms of upcoming tools solving for the presented challenges as follows:
Developments in the preparation of testing scripts and the corresponding data sets are happening on three fronts (Gherkin, ReadyAPI, TestIM, Cypress, Virtuoso, TestRigor, Katalon, Leapwork, Muuk Test, etc.):
- No-code/Low code or simple language tools (Gherkins based tools for example) to allow even non-developers/testers to perform user behaviour driven testing.
- Automation in creation of test cases through recording and parametrization.
- AI tools to learn from the existing test cases to create smart assertions.
To enhance the scope of the test cases in the CD pipeline (Meeshkan, Mesh Dynamics, Loadmill, Speedscale, ProdPerfect, TestRigor, etc.):
- Test pool creation by collecting each of the cases conducted by the developers and building an extensive test case data set collection.
- Recording and employing production data to create test cases (for front-end, API and E2E testing) which are close to real-world conditions. Dealing with privacy and data-sensitivity is fundamental for adoption of production led testing.
- Maintenance and update of testing scripts (Yourbase, etc.):
- Tools that can identify changes — say a particular modified label (in the UI) in front-end testing and automatically update the testing script to accept the change. Same holds true for API and E2E testing as well.
Communication and collaboration between team members in testing:
- Platform tools allowing both developers and testers to simultaneously build, monitor and analyse test cases coupled with distributed tracing can help in improving quality of test cases as well as identification and delegation of bugs.
Managing Dependencies and Statefulness of Databases:
A bottleneck which has been hindering the adoption of extensive (close to real-world) testing in the CD pipeline is the management of the Database ‘state’ and the script dependencies. For the test results to give a certain outcome, the state of the data stored on the database servers need to be appropriate. Use of production databases for this purpose is unfeasible as a single bug might corrupt the entire database and bring the application down.
- When a company wants to run staging tests, it provisions a database server, populate it with the relevant data, run the tests, and iterate — all of it manually. Practically, this can’t be done every time a code is committed by a developer and integrated with a branch. This is one of the biggest deterrents in adoption of continuous testing.
- As a code evolves so does its dependencies. Typically, the dependencies are listed in environment files when handed over to testers who create the testing environment accordingly. However, any mismatch in environment files or their versioning results in avoidable iterations and a waste of time.
A few upcoming trends attempting to address the above issues are:
- Creation and placement of listeners across the microservice/API network which map the input and output at each point to be able to mock other service/API calls in testing. Here, a database is neither created nor simulated, and yet the statefulness requirement is taken care of.
- Creation of a layer over the database which records and maps the input/output of a database call by a microservice/API. The recorded data can be used to simulate sections of databases efficiently and autonomously.
- Use of synthetic data generators deploying various datatypes to model and create production data with consistency (input/output) and dependency linking (between tables) whenever required.
- Tools to automate recording of dependency changes and check for the same (including their versions) while configuring the testing environment.
Creating efficient testing environments:
The cost of delayed identification of bugs increases exponentially with the growth of the application. Therefore, it’s only obvious that extensive testing is performed as early as possible in the life cycle of the code. However, the provisioning, configuration, and maintenance of testing infrastructure (servers and databases) is a time-consuming process resulting in companies ignoring these tasks as much as possible.
- Servers and databases are allocated, relevant libraries (docker, java, etc.) are installed, services are deployed along with the linking APIs, and finally the testing scripts are run. Therefore, such testing is typically done only after a significant update in the application.
- Configuration of the testing environment needs to be maintained to support the changes in the libraries utilized by the services or if any new microservice in a different language is deployed.
A few trends addressing the above issues are:
- Tools managing an alternate continuously run test server(s) running the entire or modules of the application for developers to deploy their code on, early in the process and perform extensive testing.
Tools to transform a local computer into a server as a part of the application infrastructure network to deploy the developer’s code and run testing scripts. An alternate is also to mock the dependencies locally and test the code.
Test Automation & Analysis, insights, and management of Test outcomes:
While there are plenty of tools for test automation, the analysis and insights drawn from test outcomes is manual (which test failed, where might the bug exist, etc.). Additionally, the market penetration of test management software is low [3] because staging tests are only performed at certain intervals and the management of the reports produced can be taken care of by the QA teams. In other words, continuous testing hasn’t become the norm… yet.
However, as progress happens across the different buckets discussed above, the need for a horizontal tool to overlook the created test cases, configured infrastructure (databases and servers), tests run, bugs identified, suggest improvements in test cases, provide role-based control to developers/testers, etc., and present the test state to the relevant stakeholders is going to arise. An inherent requirement for any tool being built along the above lines in the domain of continuous testing is to provide integrations with the versioning platforms, CI/CD pipeline tools, cloud servers and storage, and any other complimentary tools.
A future where a single code commit or even a small update in one of the microservices is autonomously reflected on the live application (having undergone a thorough diligence) is going to reduce the development cycle time significantly. And the developments in the testing domain along the lines discussed above is going to be a large step towards achieving that vision. In conclusion, the power of continuous delivery can only be unlocked through the adoption of continuous testing.
If you are a founder building in the DevOps/SaaS space, we would love to hear from you. Please feel free to reach out to Kanishk to brainstorm, discuss developments or opinions at kanishk@pravegavc.com
References:
