The Anatomy of the DevOps Pipeline

Breaking down the complexities of a CICD pipeline

Photo by tian kuan on Unsplash

DevOps and automation have been reshaping the world of software and disrupting the organizational cultures by enhancing the collaboration and integration between two teams to produce software faster and more reliably. In this article, I will try to explain the DevOps pipeline for a product with the details of technology implementation.

Goal :

In one of my recent projects, I got an opportunity to set up a DevOps pipeline for big data products that are continuously getting improved with user feedback. These new improvements should continuously reflect images that are getting generated.

To automate the CI/CD pipeline so that changes are continuously reflected in product images (Docker, AWS ec2, Virtualbox, and Vmware) we have implemented the below solution.

Challenges :

As DevOps provides a vast amount of open-source tools, it is a challenge to choose the right tools for the implementation. By considering various factors we have implemented a solution by using the below tools.

Tools :

• Build and Deploy: Jenkins (Version : 2.89.4)
• Scripting: Shell script and Python.
• Various machine images: Packer (Version : 1.2.1)
• Virtual machine environment workflow: Vagrant (Version : 2.0.2)
• Version Control System: Git (Version 2.16.2)
• Configuration management: Ansible (Version 2.2)
• Repository hosting service: Atlassian Bitbucket

Implementation :

1. We have set up Jenkins as a single node server. We have been using bitbucket as the VCS for our organization which made it easy to integrate with jenkins server.
2. Jenkins server has been exposed and internal IP and port has been forwarded on the firewall so that it is accessible on the external IP. Internal IP has been made static on the basis of mac id.
3. SSL is implemented in jenkins with CNAME (https://www.mycompany.jenkins.com).

4. In hook option of bitbucket repository Jenkins URL has been given and in jenkins maven project settings bitbucket URL has been given. We have implemented jenkins project on master branch. Generally you can decide branching strategy depending on your organization requirement.

5. Now whenever the new changes are getting pushed to master branch it will trigger the Jenkins build on Jenkins server.

6. On Jenkins server all the required softwares such as maven, jdk 1.8, docker, packer, vagrant, aws-cli is installed. You can manage and keep these configurations for jenkins server through Ansible or SaltStack (depending on your taste). Keep ansible playbooks and highstates of salt as a backup for cases of jenkins server failure.

7. If the jenkins build is successful jenkins will trigger post build jenkins script which will basically pull the base images from vagrant where all required softwares are already installed.

With the help of shell scripts required jars and environment variables will be replaced in the image. In this way we can generate both virtualbox and vmware images.

8. For docker the same strategy has been implemented where base docker image is created and stored at docker hub. The image is being pulled once the jenkins job is successful and all the jars and environment variables and required ports will be exposed through shell scripts.

After that image will be tagged with the version and pushed to docker hub. This way we are saving lot of times for all the software installation required and we can get images fast.

9. For the AWS ec2 images we have used packer. With the help of packer we can create our custom ec2 image and push to aws account.

10. According to your requirements for code quality you can implement sonarqube.

Conclusion :

There are technical benefits:

• Continuous software delivery.
• Less complexity to manage.
• Faster resolution of problems.

There are cultural benefits:

• Happier, more productive teams.
• Higher employee engagement.
• Greater professional development opportunities.

And there are business benefits:

• Faster delivery of features.
• More stable operating environments.
• Improved communication and collaboration.
• More time to innovate (rather than fix/maintain).

About the author

The author has over 5 years of IT experience in the areas of design, development, analysis and implementation of DevOps tools and technologies. He thrives on innovation, and loves the process of building on ideas. When he is not coding, he dabbles in photography, travels the world and enjoys adventure sports.

I look forward to receiving your thoughts on today’s blog. Write in to me at swapnil.patil1682@gmail.com.