CPT : Continuous Performance Testing

Continuous Integration and Continuous Deployment (CI/CD) is a buzzword these days. Setting up a CI/CD pipeline seems to be a part of every project plan and life cycle management. This is a very good practice. It reduces integration time and makes the the application time to market shorter and faster.

But the question is does Performance Testing fit in the puzzle for developing fast to deploy, high performing application pipelines? The answer is yes and no.

To understand the answer to this question, lets first analyze when should performance testing be executed. Performance testing must be planned from the application design phase. But it gets executed towards the end of the development phase. Typically at this stage, it becomes a bit difficult to make any major changes to the code design if there are any issues discovered in performance testing. Performance testing is a non-functional requirement. Performance testing should be done on a stable version of the application, one which has no major defects, for predictable results.

So, if performance testing is included in the devops process early on, possible performance issues can be caught and addressed early on. But this is not as easy as just including the scripts in the devops pipeline. A performance environment has to be readied and kept isolated. Everytime the build is generated and passes the CI/CD process, it has to be deployed in the performance environment as well. Now, if each build is stable (lucky you! ), the performance numbers would be a good guideline to know how the application is doing. But typically each build would have some new feature or development change, so the performance script would need to be updated. It would also need to be tested. And then after this, the deployed build will have to be (performance)tested, which kind of defeats the whole automation process.

So, does that mean you shouldn't include the performance testing as part of the CI/CD? Of course you should. The above process also works for stable applications which have just a few minor changes in each build. But the key is at which point in time should it become a part of the CI/CD. Typically, at the stage of feature completion. At this stage, the build is stable, the performance scripts are stable and enhancements can be done easily. That way, at the end of the release cycle, there wont be any surprises on what the application performance numbers are and the efforts towards tuning and fixing performance can be in a more definitive direction.

So choose the approach based on which stage your application is. You can go with either one. Remember, its better to know the performance of the application, bad or good, than have it as a surprise in production.