One Unexpected Hump That My Team Had to Overcome with Continuous Delivery
This is part of a blog series of Improvement Interactive’s journeys in enterprise application development. Improvement Interactive develops large, complex systems for a variety of clients. Today’s installment highlights one of our teams and their implementation of continuous delivery.
I recently joined one of the teams at Improvement Interactive which has been improving the systems and continuous delivery processes for one of our clients which is a large multi-national customer. Improvement Interactive has worked with this client for over 15 years. The applications our team builds for them is a large and complex solution which handles over $21 billion dollars in transactions per year.
For this application, our development team wanted to implement popular continuous delivery patterns such as canary release and microservices in order to deliver value faster and to reduce the risk associated with deployments.
We reached a place where our microservices architecture and continuous delivery pipelines were in place. It was time to realize the benefits of our efforts toward continuous delivery and we hit an unexpected hump. The development team and our client got nervous. We realized we were stalling on execution.
It became apparent that we were stalling because of the power of continuous delivery. The client and the projects development team began to run through the implications in their minds. We could literally click a button, and our latest version would be pushed to production instantaneously. In our heads, it seemed like a lot could go wrong such as harmful bugs being deployed, or entire services crashing. What if it all blew up in our face? If something went wrong the impact on the business would be huge.
Both our client and the development team needed to begin relying on the system that was built. We had to commit to deploying as often as possible. Our development team and the client had to remember the hours spent building a strong continuous delivery pipeline complete with automated tests and monitoring systems. Our team had to remember that the whole point of continuous delivery is to reduce risk. The purpose behind implementing these continuous delivery strategies was to deploy smaller changes because smaller changes mean smaller risk.
It is a big switch going from large releases for a monolithic application to smaller, more frequent changes for a componentized system. In the past, the team for this project worked towards a release date that was weeks or months in the future. In order to transition, we had to remove our previously set release date which was weeks into the future and agree to begin releasing immediately.
Additionally, we used a microservices approval workflow and a UI from Cycligent (full disclosure: Cycligent is a sister company to Improvement Interactive). This allowed our clients to review and approve changes ready for deployment in each stage of our deployment pipeline.
A Look at Our Microservices Approval Process
Here is a closer look at the microservices workflow and the “Microservices Approval UI” which is helping us to move development forward.
Our application is split up into various microservices, each with their own Git repo. When a developer finishes changes to a project, they push to a branch on the GitHub repo which is named after them (tyler_dev, for example.) The Cycligent build server detects this, and proceeds to:
- Make sure their code merges without conflict
- Check code quality metrics using the configured code inspection tool (in this case we are using SonarQube)
- Run the automated test suite for the application
If any of the above checks fail, it will abort the build and the merge, and inform the developer about changes they need to make before their code will get deployed to the Dev version. If everything is good, the code the developer pushed gets merged into the Dev branch, and then deployed to the Dev environment running in the cloud. (Note: Any version can be set up to automatically deploy, in our case it is Dev.)
Also, after the push is found to be good, the microservice that was pushed will show up in the Microservice Approval UI. My team uses the following flow of versions: Dev -> QA -> Candidate -> Canary -> Prod. The microservice the developer pushed to will show up as having pending changes from Dev in the QA section.
Our client has access to the Microservices Approval UI. Whenever our client wants, they can go to the Microservice Approval screen, see the changes that are pending, and approve which changes should be deployed and to which version. The client has full control, and as soon as they click “deploy”, a rolling deployment to the cloud servers starts immediately. As it is a rolling deployment users continue to use the application uninterrupted.
If there is a need for a hot fix then developers can push with hotfix in the name (like tyler_prod_hotfix). Our microservices approval process allows for the hotfix to skip over the normal version flow, and will appear right way in the downstream version for approval.
As a result of implementing this microservices approval process, a sense of control is reestablished and there is confidence when it comes time to deploy.
Non-Technical Hurdles in Continuous Delivery
Admittedly, the most complex aspects of continuous delivery are technical in nature. However, the cultural and not-so-technical hurdles are very real too. To those considering continuous delivery for your organization, know these hurdles can be overcome through teamwork, solid planning and most importantly, taking action.
Also, check out our continuous delivery tool at Cycligent.com. We have found this tool, developed by our sister company, to be most helpful. Not only does Cycligent allows us to deploy and manage microservices, but also support multiple simultaneous releases including canary release. Cycligent also builds an n-tier architecture for us complete with workers, long workers, load balancer, and much more. It makes deployment, configuration, and management simple, and yet is a powerful infrastructure solution for our complex, enterprise applications.
About Improvement Interactive
The core Improvement Interactive team has worked together for over 20 years. We have a proven track record of delivering enterprise software solutions to our customers. Improvement Interactive combines software development and process improvement to provide solutions to your business. Improvement Interactive stays with you for the long-haul, helping you to manage change and achieve success. We integrate scalable, secure technology into your processes and existing IT environment.
One client manages more than 14,000 units and more than $20 billion in transactions annually with an Improvement Interactive built system. For another, we built a worldwide training management system with funds collection spanning 40 countries and six continents.
Improvement Interactive is about business. We use technology to improve your processes.