The Journey to CI/CD, Part I
Solace PubSub+ Cloud is a SaaS. We have been building it using the sprint-based scrum process. There is nothing wrong with the process which basically is a miniature water-fall process. The process itself is proven. The team is comfortable with it. Yet there is something not quite right with it. First of all, the process creates an artificial separation between feature development and feature deployment. Over time, such separation creates the disparity between development environment and production environment. It is the source of many mishaps. Secondly, critical production issues cannot be addressed in a timely fashion since it is a sprint-based process. Finally, we cannot and do not want to build features based on our own imagination and assumption. It’s unrealistic, unnecessary, and sometimes harmful, but we shall be able to build something, anything, really quick. The sprint-based process cannot give us that.
To address these issues, we decide to get onto the journey to continuous integration and continuous deployment (CI/CD).
Like in any process change, the key factor to the success is people. They need to buy into the idea and they need to change their mindsets to be compatible with the new process. Surprisingly, almost everyone on the team agrees this is the right thing to do. The reason is probably that CI/CD is a relatively mature process with many successful cases. Anyways, it was one less problem to solve. As for mindset change, it’s going to be a lengthy process, constantly communicating a consistent message is the way to go. Another way to solve the people issue is to get the people out of the way, i.e. building as many process rules as possible into the CI/CD pipeline. It’s much easier to let the pipeline fail a pull request that lowers the code coverage than asking a person to do the check.
Mindset Change
- dev team is dev + devtest + QA, i.e. quality owner
- “pull” instead of “push” in term of what to work on next
- smaller changes
- one story a time
We know that we cannot achieve continuous deployment on day one. We divided the process into 3 stages. We set out the target date for each stage, we are committed.
Stage One: Continuous Integration
- Tests are continuously and automatically run over changes, failure detected, feedback is sent back to the change owner as early as possible.
- Still “release” based scrum process.
defining release content => develop => test => release to production
- set the date of the last release rolled out in this way.
Stage Two: Continuous Delivery
- Full pipeline test automation
- “trunk” is always healthy and ready. A “load” could be cut at any time anywhere.
- Kanban process, story-based small changes
New dev and release model: Picking a story => develop => test => release to “trunk”
- Release from “trunk” by choosing a point of time including desired change set to production
Stage Three: Continuous Deployment
- Every change committed to “trunk” will be picked up by production.
- Feature switches, schema change, blue/green deployment, hitless upgrade, rolling upgrade, deployment rollback, etc are the problems to solve.
