Development CI Application Using Application CI Development
Note: this is a palindrome
Bear with me, it will all make sense.
Designing a CI/CD pipeline for a brand new microservice application might sound like an awesome task at first, but a deep dive into the microservices architecture, integrations, 3rd party components, and the cloud will make you think differently. That was our starting point over a year ago when we began to develop CloudShell Colony.
The complex requirements, both from a technical point of view and from the organizational level, required a solution that would be scalable, easy to operate, and extremely reliable. We started with a small number of microservices, a few development teams, and numerous requirements for integrations with 3rd party applications and services. Our main challenge was getting all of our microservices set up on the 3rd party’s cloud-based VMs in a way that is repeatable, easy to change, and customizable.
Our product, CloudShell Colony, is a SaaS-based Environment as a Service (EaaS) platform. It connects to our customer’s public cloud and automates the provisioning and decommissioning of applications and infrastructure environment deployment throughout the release pipeline, enabling users to focus on business goals rather than on the cloud provider itself.
We need environments throughout the software development life cycle. Environments are based on building blocks and allow any person in the cycle to developer, test, integrate, demo, and deploy the application to production. Each environment is comprised of slightly different building blocks and configuration sets and will allow for different views of the application.
The solution: dogfooding. Why not use our own product to deploy our product?
In short, we will use our own set of EaaS solutions to develop, test, and deploy the actual solutions to production. Following this single guideline will assure that we have a production-ready product that is tested and customer-ready.
In the past 12 years at Quali, we have seen many of our customers struggle with testing their on-premise and SaaS applications the way “their customers will use it.” Designing a CI process that will be automated, fast, and reliable is hard enough for most of them, and getting to a point where the CI is covering the customer use-cases sometimes looks unreachable.
In our CI process, we build our code and publish our artifacts so that they will be available to the rest of our CI process. Those artifacts are then used to set up an environment using our CI Development tool; The tests we run on this new environment are creating new environments that we test to see if they are deployed as intended — that’s the purpose of our tool.
To get more technical, we use Jetbrains TeamCity on all of our automation builds, and the CI artifacts are saved straight to an S3 bucket. The magic step of our CI is to start our product by using our product.
- This could get confusing, the instance that starts our product for testing is called Deedee
- Dexter is the production instance that started Deedee
- When the new artifacts are tested and found good, we use them to start a new Dexter
- Once we have new features that are important for launching a new environment, we use Dexter to launch a new Deedee with those features, so that it could start new testing environments in our CI
The testing environments, we call them Mandark, start the integration tests that use that Mandark instance to kick off new environments.
For our production environment, Dexter, we use our solution’s “Production” feature, which is a Blue/Green deployment mechanism. Whenever the CI passes and we have validated artifacts for deployment, a new Green is started and then tested again since it’s on the production database and new issues that were hard to test may be caught. When the Green tests (mostly web automation tests) pass, we are ready to issue the promote command, this command is handled by our CI tool; it moves the traffic from the Blue to the Green environment. This is done via ALB (Application Load Balancer).
After the promotion command is completed, the old Blue environment is terminated, and the Green is the new Blue. Of course, we can roll back the Green if the tests fail, or manual test is unsatisfactory, then a new Green will be started after changes are made to fix the issue and all the tests run. From the new code checked in, to the production having that change, we have our CI that runs for 2–4 hours. Each time a new feature is added to our product, it can usually help us better our CI process, so we upgrade our Deedee and start using it.
In conclusion, dogfooding helps us create a better product. Since we are our own customers, we find the issues our customers would encounter prior to them encountering it. Yes, it was hard at the beginning when things didn’t yet function as we’d hoped, but the product matured, and I can’t think of a better way to implement our CI process.
Tomer Cohen — software skeptic
Colony core developer
