Scaling your CI/CD
Continuous integration/continuous deployment (CI/CD) is a set of practices that enable development teams to merge code changes frequently into a central repository and confidently deploy them for users.
As part of our hybrid cloud journey, the IBM CIO organization discovered that a key component of our transformation is safely building and deploying applications at scale. Therefore, we’re developing a common organization-wide CI/CD solution built on Red Hat’s OpenShift Pipelines. This common pipeline approach allows our organization to build on previous work to scale best practices across the organization. But why are we doing this? I’ll lay the foundation for why not just CI/CD, but a common CI/CD solution is beneficial.
What is CI/CD?
CI/CD is a two-step process that removes friction from the development and delivery process using automation. CI is a practice where the development team frequently commits changes into the shared repository and then leverages automation to check that the code still works as expected. This happens by having each commit trigger a build and a series of automated tests to verify the application’s behavior and the clean integration of updates. This enables the application team to deploy code frequently, which leads to continuous deployment. CD automates releasing an app to production. Because it is automated with no manual gates, CD relies heavily on well-designed test automation. Once the application passes all tests, a developer’s change to an application can go live within minutes
How is CI/CD implemented?
CI/CD automates the process of integrating, releasing, and deploying software while removing roadblocks. This is done with smaller code changes, continuous testing, real-time feedback, faster releases and improved customer satisfaction, and reduced costs.
• Smaller code changes: CI/CD encourages developers to integrate small pieces of code. Smaller code changes are simpler and easier to handle than trying to integrate a large amount of code. This is especially true when a large number of developers are working on the same code base.
• Continuous testing: Building on smaller changes, continuous testing allows these smaller pieces of code to be tested as soon as they are checked into the repository so that developers can see and correct problems quickly.
• Real-time feedback: CI/CD is a great way to get continuous feedback not only from end users but also from the developer team to improve team transparency and accountability. This real-time feedback increases visibility into problems with the team and encourages responsible accountability.
• Faster releases and improved customer satisfaction: CI/CD pipelines provide a means for consistent builds, tests, and deployments, enabling errors to be detected faster. This allows teams to safely release code faster and with new features and bug fixes. Development teams can meet users’ needs through regular (even daily!) updates and responding to feedback with rapid, high-quality changes.
• Reduced costs: CI/CD pipelines help catch bugs before they reach production, significantly reducing the costs of addressing them. Deployment automation also reduces the chances of human error affecting deployment.
A security example
In late 2021, a zero-day vulnerability was discovered in the popular logging library Log4j 2. This vulnerability enabled attackers to use a simple string of text to trick a system into requesting and executing malicious code. This vulnerability was rated at the highest possible CVSS severity level and estimated to affect over 90% of enterprise cloud environments.
In addition to the obvious security issues, this incident illustrated that many organizations do not know what components make up their systems or, in some cases, even what systems they are running. It demonstrated the need for organizations to understand what their software is built from and any additional security requirements it demands.
You may be wondering what the Log4j 2 incident has to do with CI/CD. This incident highlighted the need for enterprises to build guide rails into their processes to help them gain better insight into their security positions, something a common CI/CD process can enable.
Security guard rails
Enterprises and chief security officers worldwide have seen recent high-profile vulnerabilities as a wake-up call to rethink security. These new approaches include:
• Secrets and credentials detection
• Open source library inventories
• Open source usage approvals
• Open source vulnerability detection
• Code quality and automated test coverage
• Static Application Security Testing (SAST) vulnerability detection
• Dynamic Application Security Testing (DAST) vulnerability detection
• Container image vulnerability detection
• Artifact signing
These new activities may require significant compliance work for development teams. If you understand that CI/CD helps team catch errors early in the process and make security part of the regular development process, CI/CD pipelines are a natural place to start, and a common CI/CD makes it even easier. With a common CI/CD in place, you can add new security tools without affecting existing development processes and developers.
This is the approach we are taking with our common CI/CD journey. By adding security to our CI/CD process, we are introducing guide rails for our developers that help them focus on delivering business value while also protecting the enterprise. Our common approach provides a process that:
• Implements our security policies, including scanning
• Reduces developer response time for issues like Log4J
• Promotes application quality and consistency
• Offloads compliance and ongoing maintenance burdens
In addition, on the CD side, we can provide:
• Production deployment approval auditing
• DAST processes
• Deployment frequency and duration metrics
• Deployment region location information
Economy of scale and friction reduction
Most development teams agree on the value of a CI/CD pipeline; however, they also may face barriers to adoption that may be related to costs, skills, or time. And as more requirements are placed on teams, these barriers can get higher.
These barriers and ever-increasing compliance requirements create friction in the development process that can reduce developer productivity. From the organization’s point of view, every team implementing its own CI/CD approach creates several inefficiencies: each team spends time creating and maintaining the pipelines, procuring and managing the infrastructure, and integrating any new tools. Combined, these increase the costs and delays in deploying features.
A common CI/CD provides a way to address these problems. A single team is responsible for creating, maintaining, and standardizing the pipelines; supporting and managing the infrastructure; and integrating new tools. For our development teams, this means:
• Greater ability to focus on business value and not process and infrastructure
• Increased productivity and satisfaction
• Accelerated application modernization
Altogether, this reduces the burden of CI/CD adoption for development teams and friction in the development process. Couple this with the ability to leverage a single infrastructure, and an economy of scale starts to emerge.
How a common CI/CD approach helps generate insights
Moving to a common CI/CD approach helps our organization collect and analyze data from the pipelines. This enables greater insight into an application’s source code all the way to deployment from a common set of tools pushing into a common data store. We’ve developed of a data lake to bring the data together and a developer experience portal to provide a single pane of glass that makes data easy to understand.
For the IBM CIO organization, our approach allows us to:
• Better understand our open source usage
• Better understand application quality
• Gain insights into our application runtimes and languages
• Reduce response time when security issues arise, such as Log4J
Conclusion
CI/CD is a set of practices that enable teams to safely build and deploy applications. Introducing a common CI/CD solution provides a means to scale the benefits of CI/CD across an organization while reducing the overall adoption effort. With such an offering in place, we can add security compliance activities to the pipelines without overloading teams. This results in not only a safer, more knowledgeable organization, but also happier and more productive developers.
Ryan DeJana is Senior Technical Staff Member & Architect, CIO Hybrid Cloud Platforms at IBM based in Boulder, CO. The above article is personal and does not necessarily represent IBM’s positions, strategies or opinions.
