Leading an enterprise to adopt continuous delivery is messy. Every leader wants to be noticed for their forward thinking. But in truth, our executives just what to measure:
- How frequently we deliver value?
- How long does it take to deliver value?
- Do our customers use/want the value delivered?
Anything more detailed is lost. Luckily, implementing a continuous delivery pipeline is easier than ever. So how do enterprises achieve continuous delivery as part of their digital transformation.
First, no two enterprises are alike…but similarities exist. I can’t pretend to capture all the experiences out there, but I’ve seen three stages that helped me define a path towards achieving basic enterprise wide continuous delivery.
Stage 1: Fragmented CI/CD — Multiple DevOps Centers of Excellence
The Situation: Throughout the enterprise, multiple technology leaders have begun their own digital transformation for their organization. Often, these leaders champion DevOps practices, automation, lean strategies, and innovation and an open floor plan. Each team implements a tool chain based on past experiences and “best practices.” The teams have begun to automate their work using open source products or have contracted out to different SaaS collaboration, CI, and security tools.
Problems: Executive leadership cannot measure success or productivity between the groups. Each measures slightly differently. Leadership believes there is merit in automating value streams but needs proof to show investors the benefits of a digital transformation. The procurement team has no idea what all these products are or what they do. There is no strategy to minimize cost of licensing or reduce time negotiating deals. All of the sudden, there are 15 different pipeline tools all deploying into production environments and the reliability teams cannot keep up with the incidents being created. Each tool requires platform engineers to maintain.
Path forward: Senior leadership needs to bring DevOps thought leaders in the organization together in a consortium. The consortium needs to articulate a set of clear goals of what senior leadership needs drive future investment. Some goals that I like are “measure technical health throughout the enterprise,” “continuously deliver code to customers,” or “provide a frictionless developer experience.” These goals don’t tell your thought leaders exactly what to measure or do, they are purposely vague to allow the “how” to extend to different enterprises and situations.
DevOps thought leaders, as part of the consortium, need to define how each of their organization’s moves forward. They need to figure out how to begin the journey to continuous everything. How they can coordinate their resources and build a structure that delivers senior leadership the data they need to justify moving forward. This team needs to translate goals into achievable steps in the form of a continuous delivery roadmap.
Stage 2: Enterprise CI/CD — Established DevOps Consortium
Situation: The enterprise has implemented toolsets to automate the building and testing of code. After code is ready to deploy, there is still a change management system that governs a separate workflow to allow developers to deploy code to production environments using blue/green deployment practices. When code is checked in, it is automatically scanned for open source vulnerabilities, code quality, and security vulnerabilities. Once it’s built, it’s packaged and tested through a variety of unit, load, and functional tests. Senior leadership is using multiple dashboards to see what’s happening across their organizations.
Problems: This CI/CD tool chain allows developers to focus on delivering code, but onboarding new applications and engineers is still a trial and error process. Pipelines run, but not fast enough. Developers need to submit tickets for a future change window because even though infrastructure and configuration are code; firewall rules, ACLs, database and other coordination is still required. Deployments are done using blue/green deployment processes, but unknown error’s with live customers still occur and occasionally require rollbacks.
Code is scanned and tested before it is deployed, but dependencies can develop security issues over time. Developers have moved on to other projects and code vulnerabilities are not fixed in expected times. When security issues do occur, auditors need access to multiple dashboards and log streams to piece together events.
Leadership is using a collection of dashboards. The data is very interesting but focused on reliability. They still have interns building excel pivot tables to explain what’s going on within their value streams and how they are delivering new features to customers.
Path forward: Stage two builds upon the cross organizational leadership consortium from stage 1. As the consortium matures, the DevOps leaders throughout the organization need to meet regularly to socialize problems their teams have. The alignment of problems and goals is the consortium’s primary focus. After they know their problems, they work together to solve the problem that causes the biggest latency. Solving even one problem together is how the enterprise starts to move forward. I’ve seen these solutions attempt to come out of one group or a lab, but it never seems to drive the group consensus that is needed to build upon for solving future problems.
Stage 3: Continuous Delivery as an Enterprise Service using Innersource Contributions
Situation: The entire organization has the ability to continuously deliver. Every code, structure, and configuration is delivered using a pipeline. We have a executive dashboard that shows how frequently we deliver value, how long it takes to deliver value, and how our customers have adopted the new value. You think you are finished?!?!
Problem: The problem is you don’t know what challenges are next. Your customers are happy, but the market is changing. Developers need to be kept interested
Path forward: The next phase that worked for me was to promote shared contributions though innersource initiatives. By sharing tool development and allowing developers to directly contribute to your enterprise tools, you shorten the time it takes to add new features to your pipelines. Eventually the community begins to want to share code. Developers begin to innovate on the internal plumbing. This accomplishes two things: reduced cost of engineering overhead and increased adoption of shared solutions from your organizational groups. Beyond sharing code, the next goal is to learn how to share development resources between the groups. The seems natural because all the developers are now speaking the same patterns, using the same shared code, and deploying on shared pipelines.
As we all know, our enterprise digital transformation is vague and open-ended. Learning and evolving is never done, it is continuous. By focusing on these three stages, adopting continuous delivery becomes a cycle of continuous learning and sharing.