Building an Efficient CI/CD Pipeline — The Airtel X Labs Way

Building and Deploying Software with CI/CD at Airtel X Labs

Airtel X Labs powers the innovation at Airtel, which means delivering consistent quality while staying agile. With the automated build, testing, and monitoring, the chance of failure is drastically reduced. Since Airtel Digital Platforms works on a massive scale, Canary releases ensure that there is zero downtime for customers and only the most stable upgrades are rolled out to all the users.

Automated runs with heavy weightage to test automation & release health automated monitoring, helps business pace up to the agility with quality. Seamless integration of Jira, Jenkins, Automated monitoring & slack helps to ensure a robust deployment pipeline.

The concept of canary deployment

The name “canary deployment” comes from a British mining practice in which canaries were used in coal mines to detect toxic gases — if something bad happened to a canary, it gave miners a sign to evacuate.

Canary deployments in software are built on the same concept — new releases are sent out in a staggered fashion and monitored to ensure that they “survive” the target audience. Process & Automation are 2 important pillars of building Canary.

Standardized processes for a robust pipeline

Standard software practices were introduced to enable automation of pipeline like:

1. Standardize branching strategy & enabling back merges from release to feature.
2. A standardized set of tools and technologies for deployments.
3. The inclusion of backward compatibility and server switching tests as a part of automation
4. The consolidation of different monitoring points into a single one.
5. A final score calculated from a combination of automation coverage and maturity of monitoring KPIs ensures quality gatekeeping.

Seamless automation with zero manual deployment

The stages within a software pipeline are automated using the following steps:

1. Build compilation and deployment: The build compilation and the deployment of code on servers is automated using groovy scripting and custom code in python on Jenkins. The inventory is picked up from a central database and the maven/Gradle builds are compiled together.
2. Test Automation: Unit tests are executed during the build compilation followed by automation tests executed after deployment on the pre-production code. The combined code coverage of both tests is calculated using a combination of jacoco and sonar. Once the automation is executed, the coverage is captured using jacoco dump MBean on JMX and writing a program to call it. Data is then pushed to sonarqube for a combined coverage of unit and test providing easy integration with jacoco with mature reporting.
3. Monitoring automation: An automated release health report needs to be generated at two points — one, as a post staggered release report to identify the issues once the product is released on a server; and two as a full rollout release report that identifies tricky problems like increase in DB load due to queries on unindexed fields.

The consolidation of system, application and business metrics in a single place can be challenging. Using InfluxDB as the time-series database for consolidation, writing python-based schedulers that poll data from the pipeline database and by monitoring KPIs of the application, a release health report is generated.

The diagram below illustrates the monitoring of pipeline architecture.

Monitoring Pipeline Architecture

4. Stitching all the steps

Finally, all the pieces of the automation are integrated at a common place and that orchestration layer is used to perform jobs. Given the deliverable timelines and the usability of the orchestrator, Airtel X Labs used Jira as the workflow manager for the complete CD pipeline. Using post functions on Jira, the team integrated it with Jenkins and the monitoring scheduler.

The Jira workflow is shown below.

Deployment Workflow

How an application is on-boarded to Canary and executed

Once the prerequisites have been completed by the application, on-boarding is done in a couple of hours with the help of a groovy script and two configuration files. These include,

- A JenkinsFile: It contains the link of the common deployment module and it checks out the code

- applicationDna.yml: It contains information about the build tool, artifact relative location, automation execution command, and the jacoco file path.

- MonitorKpi: It is used for generating the release health report of the module and is configured with the data required to be fetched from InfluxDB, comparison type for the metrics and a margin until which it can hold the value before failing.

A WebHook is configured on the code repository for deployment on feature branches in a lower environment. On commit to the master branch, it initiates the complete production deployment pipeline.

Continuing the automation journey

With automation and standardization of deployments, the time to market for a release is significantly reduced and better quality products reach the market. An efficient CI/CD pipeline with canary deployments strengthens quality checks, simplifies the deployment process, and allows developers to spot issues before they become outages.

At Airtel X Labs, this is just the start. We will continue to leverage our learnings to provide cybersecurity integrations with CI/CD pipeline, including DML as part of automation, communication channel integration, generalize data conversion plugins and more, in the future.

What’s next for the readers

This is a part of a series of blogs from Airtel X Labs highlighting the innovation used on our production floor. In the next blogs, we will explore implementation and automation at each stage in greater depth, and outline the generic code that can help you build your own CI/CD pipeline.

Thanks to Gabavarun & Nikhil Sharma for their contribution to this project.