Digital Transformation: Ten Components of a DevOps EcoSystem Foundation
Introduction
Setting the Stage
Driving a global scale digital transformation is a complex journey. Execution of digital transformation at scale requires comprehensive planning to drive organization and cultural changes, but concurrently requires an automated framework to safely drive the same transformation at scale. When considering the adoption and implementation of a next generation operating model, the manner in which this model is deployed will mark the long-term critical path toward consistent adoption.
Aligning Transformation and the DevOps Ecosystem
One critical component when driving transformation at scale lies in the ability to ensure consistency of the journey and adoption. When considering adoption in the context of a comprehensive digital transformation, this must be shared at as an overview program campaign, and at the same time must also be adopted tangibly via a suite of concrete tools (DevOps ecosystem). This will drive both the ease and consistency of digital and DevOps adoption.
Below is an overview of the ecosystem journey, and in the following sections, we will review the overview, rationale, value and challenges of each component.
Driving the DevOps EcoSystem “as a Service”
One important note. As an organization embarks upon the journey to drive a comprehensive delivery (DevOps) ecosystem, a central tenet is to ensure that each component is offered within the constraints of an “as a service” model. This is important as an as a service model (or aaS model) lowers the bar to tool and ecosystem adoption, by allowing teams to consume services in the easiest manner possible.
Note: In most cases, these tools are built centrally. However, the manner in which these as a service tools are designed within the context of an Open and/or InnerSource model will be covered in future posts.
DevOps EcoSystem Baseline vs. Exhaustive
The following is in no way meant to serve as an exhaustive list to a DevOps ecosystem, but rather meant to serve as the minimum foundation over which the balance of components will be added over time. That is, while components such as testing tool suites, configuration management engines, and image repositories are all critical to a mature DevOps ecosystem, the description contained within this discussion forms the foundation that must first be established. Once that foundation is established, the DevOps ecosystem may continue to evolve and mature over time.
Component One: Single, Global Process Definition
Single, Global Process Overview
Within the context of digital delivery, the entrant point along the delivery value chain lies in defining a single, global delivery process for the organization. While this may not mean that every geography or business unit will uniformly adopt the process identically, it does infer a minimum level of consistency. In adopting this consistency, this now allows for a definition of a single end-to-end delivery process, that can then be defined and automated within the context of a common DevOps ecosystem.
This process must be defined in a manner that is universally accessible to the entire organization. In this way, the process becomes transparent, shared, and common to the wider global group. This drives a common understanding of a single method that is open to sharing, common execution, and constant refinement.
Single, Global Process Rationale
The single, process definition is a an anchor upon which all subsequent automation implementation steps align. For this reason, it sits in first place, and must be generally defined ahead of the balance of automation steps. It remains important to recognize that these steps are not set in stone, nor is this meant to serve as the definitive process to drive the organization forward for all time. Rather, this process will provide a foundation upon which to build the balance of automation steps, and iterate upon these steps over time, refining the process to maintain parity with the requirements of the organization and the wider marketplace.
The first step within a digital transformation is to establish a common operating model and end-to-end process definition. Without that, the technology components will have no foundation upon which to rest. (Single, Global Process Definition)
Single, Global Process Transformation Value
The importance of establishing a single, global process lies in the ability to drive consistency in the operations and delivery methods of the organization. By establishing a single, documented process (as distinct from methodology), it gives insight into the expected steps and outcomes required, driving consistent compliance across the organization´s various business units and disperse geographic regions.
Challenge. Establishing a single, global process is complex to define, and even more complex to implement. Driving consensus, and at times simple definition, across a disperse organization is complex and fraught with competing business unit and geographic priorities.
Driving a single process is iterative by nature, and this level of uncertainty is often hard to adopt without senior management buy-in at the highest level. With that said, the benefits of a single, global process and the ability it provides to automate that same process returns dividends to the organization in productivity, velocity, and quality of delivery.
Component Two: Global Planning & Definition Tool Suite
Global Planning and Definition Overview
Driving common planning, and dependency management inside a single tool suite is important within the context of a cohesive digital transformation. Of course, one single tool suite is not required in order to be successful in an ability to plan across a global organization. That said, having a strong, enforceable set of processes within a limited set of application suites assists in driving consistency of planning, adoption, and dependency management.
A single suite, in many cases an agile delivery tool, allows for all units and geographies to plan collectively, using a common methods and understandings of delivery expectations.
Examples of tool sets commonly adopted in this suite would be tools such as Atlassian´s Jira, Computer Associate´s Agile Central, or VersionOne.
Global Planning and Definition Rationale
By establishing a single planning and definition product tool suite, the organization may now take the first steps toward driving a more comprehensive and cohesive process. In adopting a single tool suite, or at least a limited set of tool suites, common planning processes may now be adopted spanning and connecting business units and global regions.
At the same time, common definition and dependency management methods can be established, ensuring more comprehensive and aligned delivery across the wider organization.
Once you have established a common delivery methodology (processes), the next step is to define a global manner in which to define, establish dependencies and progress delivery work definition. (Global Planning & Definition Tool Suite)
Global Planning and Definition Transformation Value
By implementing a single process suite within a common tool suite, globally available, the organization takes the first steps toward driving cohesion between the operations and processes across business units and geographies.
This common implementation drives delivery definition visibility. At the same time, this planning tool suite will also enable common visibility of dependencies across business units and geographies. While this will not solve the complexity of delivery within an large-scale organization, it will begin the journey to streamline visibility and dependencies through common planning ceremonies that can now leverage the power of common visibility through a single (or limited) tool suite.
Challenge. Driving comprehensive migration of all planning activities to a single tool suite will take time. In nearly all cases, planning occurs in each business unit and geography already. In some cases, these are rudimentary tools, and in other cases, extremely sophisticated use cases. This journey must be conducted over time. While the long term alignment to a limited tool suite will cost areas in the near term (the most common objection), the benefit of common visibility and dependency management will far outweigh those costs over time.
Component Three: Global Communities
Global Communities Overview
Once the early stages of establishing a common delivery methodology and process have accomplished, and a common tool suite has begun to be used across the global organization, it now becomes important to plant the seeds for the long-term global collaboration that will be required in order to drive true, global cultural change.
Global communities, with a centralized mechanism to unite the teams, including communication tooling is critical to drive this next cultural shift, and drive visibility of digital change across the global organization.
Global Communities Rationale
As teams begin to drive change within their individual teams, driving the adoption of a common definition tool suite, and at the same time adopting a new paradigm for digital delivery (methodology, operating model, new processes), it will become increasingly important to know that they are not alone in the journey, and have the support of a global organization, as well as their peers in the transformation upon which they have embarked.
Once an organization has begun the adoption of a common manner of delivery (methodology, process), and embraced the usage of a common definition and delivery system, it will now become important to adopt a mechanism of global community. This community will help ensure a global vision for the transformation and usage of these common tool suites and practices, while at the same time helping to drive local adoption flavors related to the digital cultural implementation.
Once the early stages of establishing a common methodology and process have begun, and early definition tool suite implementation has begun, it now becomes critical to establish global communities, made up of global and regional team members to drive the transformation into the fabric of the organizational culture. (Global Communities)
Global Communities Value
The value of establishing global communities cannot be underestimated. These communities, made up of a global team comprised of global and local business unit and geographic team members, are the catalyst that will drive the transformation to the next level.
It is during this early stage, before the entire ecosystem has been rolled out, where the seeds are planted to begin to form this group, one that will continue to grow and evolve with the transformation over time.
To dig deeper into this theme, please see the step two (setting up the catalyst organization) of the article Digital & DevOps Transformation: The First Three Areas of Focus.
Challenge. The challenges in setting up a global community are not small. When we think of global communities, we envision the exterior items of note such as meet-ups, devops days, etc. However, establishing an effective global catalyst team, while at the same time establishing local teams driving a cohesive transformation message should not be underestimated.
This involves changing the very fabric of the culture and the manner in which organizations interact, and this affects the organization, operations, processes, and tooling on a global scale. Effective leadership is required to drive this both at the global level, as well as within each business unit and geography where the transformation focus has begun.
Component Four: Common, Global Repository
Common, Global Repository Overview
Once a common planning tool has been established, the teams will now begin to build individual components related to the definitions driven through the now-global planning processes. These technology components must have a common place where they can be stored. This is the global repository.
By placing all of the components built by our global teams (and also used for local-only components), we are able to now execute delivery against a single, common inventory of components, and understand holistically the delivery of the global organization.
Common, Global Repository Rationale
Implementing a single, global repository now allows for a simplified inventory of all of our application components built within the delivery organization. This information can then be associated back (automated) to the inventory of definition held within the global definition tool suite (component two).
By having all components within a single, global repository, the capability to now align all dependencies in a much more intentional manner now becomes a more realistic possibility. While this in no way magically aligns all of the complex work in the development organization, it does give common access and visibility to components for both dependency management and component reuse.
A few examples of suites used to create a common global application are: github, giblab, bitbucket, subversion. This article will not discuss the pros and/or cons of a chosen repository solution, but is rather meant to review the overall capability vs any product directly.
As the product definition, and common dependencies are established, the next important step in the chain is to drive a single, common place where application/code components can be housed, and dependencies aligned in an automated manner. (Common, Global Repository)
Common, Global Repository Value
By establishing a common, global repository, the organization will have the ability to benefit from global capabilities that can be used at-scale, but the truest lift will be the enhanced level of communication across our global teams. The gains realized will be observed minimally as the following:
- Components can be linked directly back to the global definition tool
- Dependencies across components can be more easily linked
- Components can be more easily identified for common re-use
- Common capabilities can be more easily shared for common contributions across an Open/InnerSource model
While those benefits listed above are several of the most important, the gains achieved by sharing globally across a common repository cannot be underestimated, given the positive synergies that evolve across global teams over time.
Challenge. As with all solutions, while there are heavy benefits associated with the global, common repository, it is not without its complexity. The first major hurdle will be, in many cases, the migration of local code repositories to the global, common repository. While much of this work can be automated, there is no doubt that much cannot, and it is not a trivial undertaking to unbuckle components out of their prior infrastructure.
Legacy components also remain a challenge, in that these often become a hybrid strategy to house greenfield and legacy code components.
The final challenge lies in regulatory concerns and the management of components globally. These must be managed within the context of the local information security and regulatory policies of the organization in each of the geographies where that organization operates.
Component Five: Global CI/CD Pipeline Pattern
Global CI/CD Pipeline Pattern Overview
Once the global repository has been established, the hooks from the pipeline to the repository can then be established at a component level. The importance of this stage lies in that now components, once built in collaboration, can be automated through the stages of environment progression.
An important note related to this paradigm lies in that we drive common deployment of our delivery components from repository to deployment. This does not necessarily refer to an end-to-end product (which we will discuss during orchestration), but rather a single, defined application component.
Automation of the application components through the use of a common pipeline pattern will given automated consistency to the manner in which applications are tested, secured, and deployed through progressive environments.
Global CI/CD Pipeline Pattern Rationale
Establishing a common pipeline pattern drives consistency within the overall continuous integration, delivery and deployment process. By establishing a common baseline pattern from which all subsequent pipeline “instances” are built, common (global) standards across testing, security, regulatory and general policies may now be established.
A few examples of pipeline solutions used in creating these patterned pipelines are: Jenkins, Bamboo, GoCD, or TravisCI . This article will not discuss the pros and/or cons of a chosen pipeline solution, but is rather meant to review the overall capability vs any product directly.
Once a common, global location has been established to house application components, automating the journey (Continuous Integration, Delivery, Deployment) from development through to production is then required to drive consistency at global scale. (CI/CD Pipeline Pattern)
Global CI/CD Pipeline Pattern Value
By driving a common template (pipeline served “as code”), common global policies and practices can now be more easily established. As we consider the importance of security policies across the organization, as well as global regulatory compliance, a common pipeline pattern becomes invaluable in this effort.
While the complexities of a global organization will necessitate some variability in pipelines by region, a common baseline template pattern will allow basic commonalities and a foundational standards that can then be applied to the entire organization.
One final note. It will be extremely important to ensure that this pipeline solution is offered “as a service.” These solutions are already complex in their deployment, and therefore driving a low bar (easy) of adoption for global teams is critical to the success of initiative over time.
Challenge. In many cases, especially when considering complex, global organizations, most organizations are not beginning their CI/CD journey from the start. Rather, in order to achieve the usage of a common pipeline, many already-in-existence pipeline must be retro-fitted and utilized in order to bring these pipelines to the global standards, and up to the “as code” standard.
Additionally, many local business units or geographies may already have standards that surpass the global template during the early days of adoption. Driving a global cohesion and local priorities will always be a balance, and creating a plan to coordinate these over time will be critical to the success of the pipeline consistency over time.
Component Six: Global Delivery Visibility
Global Delivery Visibility Overview
As the early components of culture and operation begin to take hold at a global and local level, the next stage is to drive visibility of that cultural shift into the fabric of the organization culture.
In many cases, the idea of a universally visible state of delivery will be a complex concept for many organizations to wrestle. That is, this information, in many cases (and, in many cases, for many years) has been locally understood, but never extrapolated to a more visible, global concept.
This fist visibility will, by no means, be universal. However, it will for the first time begin to drive a common understanding of what continuous delivery, integration and deployment means to the organization. By making these metrics universal, and visible to all, a healthy fact-based debate can now begin across the organization.
Global Delivery Visibility Rationale
As Dominica Degrandis mentions in her book, Making Work Visible, one of the most important things we can do to help organizations and individuals improve efficiency through visibility.
When we create tool suites that allow us to drive common visibility across our delivery life-cycle, the organization can then commence a meaningful dialogue around those delivery expectations surrounding velocity and quality (including concepts of reliability).
As the early components of the DevOps ecosystem are put in place, the next important step is to drive visibility into the system to drive maturity, through a rich dialogue that comes by making our delivery status far more visible. (Global Delivery Visibility)
Global Delivery Visibility Value
As work within the early areas of delivery definition, delivery housing (repository) and continuous delivery (pipeline) progress, these components can now be made visible to the teams and leadership in a very tangible way. By doing this we now create a common definition and a facilitate a richer dialogue at the enterprise level to drive consistent improvement throughout the early stages of the delivery life-cycle.
Challenge. There is no single way in which to monitor components, code re-use, velocity, or quality within an organization. When building visibility into the system, it remains important to keep early definitions generic (e.g. velocity = time from final merge to production deploy), as the debates that these definitions will generate will be impactful to teams, as no two team application components will ever be the same.
It will be additionally important to culturally ensure that this tool suite is not used to show why a team is better or worse than another. The natural tendency to drive healthy competition is an important one, but also one that must be managed within the context of the organization´s culture to ensure that visibility is not used in a counterproductive manner.
Component Seven: Global Product Orchestration
Global Product Orchestration Overview
Once the organization has begun the basics of common planning, placing components in a common repository, and begun to automate those components along a common template (patterned) pipeline, it will now become critical to begin to organize and orchestrate those components within the context of a product or solution context. That is where the orchestration engine enters into the ecosystem.
Global Product Orchestration Rationale
Orchestration allows for complex products and business events to be orchestrated across the defined work and components held in the repository and continuously integrated via the common, global pipeline solutions.
By establishing a common orchestration layer, this allows for business events, complex components, security and release context to all be aligned into a single framework to ensure components are deployed together with all appropriate dependencies, both technical and business driven.
A few examples of orchestration solutions used in creating these orchestration patterns are: XebiaLabs, Electric Cloud, or CA´s Continuous Delivery Director. This article will not discuss the pros and/or cons of a chosen orchestration solution, but is rather meant to review the overall capability vs any product directly.
As automating product components along a single pipeline (CI/CD) becomes standard within the organization, the next evolution will then be to drive product centric deployment, orchestrating across components (pipeline), including business dependencies to drive a more cohesive, business centric, deployment criteria. (Global Product Orchestration)
Global Product Orchestration Value
By establishing common orchestration, application components may now be organized according to business, operations, and technology dependencies. These dependencies are recursive, and can therefore ensure that all components are built in such a way as to ensure that any and all dependencies are respected and upon deploy.
The full ecosystem (along with its dependences) are driven through their respective CI/CD pipeline, and deployed as one common product definition set.
Orchestration allows us to now view products in a holistic manner. In some cases, these may retain a system context. The key benefit to orchestration lies in that it allows components to be managed within a wider context of a product (including business, security, and technology constraints) or valued service, and deployed within that same context.
Challenge. Extracting a product definition out of the CI/CD layer is often complex, especially for many organizations who have only recently begun to embrace the concepts of “continuous.” In building this layer, moving these components to this new context (and in most cases infrastructure) is complex. This will require mapping dependencies not previous envisioned. However, while not an easy task, the order created by driving this product centric view gives a higher level of visibility (likely never-before-seen) within the deployment and release life-cycle.
Component Eight: Global Policy Enforcement
Global Policy Enforcement Overview
As the organization drives forward along the maturity curve of continuous integration, delivery and deployment, it now becomes increasingly important to safeguard the delivery system. That is, it remains paramount to ensure that appropriate security measures of observed, regulations are followed, and organizational policy is protected.
In doing this, we create a central policy store which will drive the definition of organizational global behaviors. Within this context, application components will be build throughout the organization´s infrastructure to ensure that these policies are observed, visualized, and enforced.
Global Policy Enforcement Rationale
The complexity of the end-to-end delivery life-cycle becomes increasingly complex when an organization begins to drive ever-increasing agility and DevOps behaviors. The speed at which application delivery is expected to deploy increases on a regular basis. Ensuring that the basic behaviors around security, compliance, and organization policy observance must never be sacrificed within this context.
It is for this reason that the policy store, and ultimately the policy enforcement engine, are built: to ensure that critical behaviors are defined, and enforced in an automated manner to protect the organization while the agile and DevOps journey drive forward.
As the wider organization begins to delivery complex software through product-centric orchestration, and the speed of delivery continues to increase, it now becomes increasingly important to establish basic global policies and standards to which all applications must adhere prior to deployment. (Global Policy Enforcement)
Global Policy Enforcement Value
By creating a single, global definition of the policies and behaviors required throughout our delivery life-cycle, we are now able to understand, view, and adapt key behaviors that required compliance at the global (organizational) level.
The importante of this lies in ensuring that we define those critical elements in order to allow application components to release. These may be security, audit, company policy, and business initiative driven. However, by establishing these, creating visibility around them, and enforcing them in a consistent manner, we are then able to safeguard the organization (quality), and at the same time drive speed (via our DevOps processes and tools) into the delivery ecosystem.
Global policy enforcement is what will allow an organization to deliver upon the promise of speed and quality in a safe manner, by ensuring guardrails ,and thereby giving our engineering teams the freedom to operate within the bounds of global established policies. (Global Policy Enforcement)
Challenge. Creating this visibility and common policy definition is no simple task. The ability to detail the end-to-end delivery process, and define the safeguards across security, company policy, audit criteria, and business rationale is a large undertaking. Arriving at this cohesive definition is an iterative and evolutionary process.
The second complexity in driving this change is readiness to enforce common standards across a global (large) organization. In many cases, individual units or areas may deploy according to local policies, including varying standards around security and threat tolerance. Driving a single standard, with a common outcome (denial to deploy, auditor warning communication, etc.) is a maturity level that will take time to achieve organization-wide.
The final complexity in implementing global policy enforcement will be the varying technologies that execute the behavior-based policies. The definition may exist, as well as the desired outcome (e.g. never deploy a client facing application that does not have, at least, dual authentication). However, implementing a common standard across new (in many cases, cloud) and existing (in many cases, legacy or mainframe) are patterns that will be defined and reused over time; however, the early use cases will be hard-fought implementations to define a reusable global policy pattern that has likely never before existed in the organization.
Note: In future posts I will dig deeper into the implementation of a global, policy enforcement model. For the purpose of this article, it is solely meant to serve as an understanding that policy enforcement serves as a key component within the foundation of a digital and DevOps ecosystem.
Component Nine: Global Transformation Visibility
Global Transformation Visibility Overview
Once organizational maturity has begun around the conversations surrounding global policy enforcement, the need to visualize these policies now moves to center-stage. Once the ability to enforce policies in an automated manner becomes a reality, now it becomes important to drive this into the culture of our leadership and delivery teams.
A key ingredient in becoming an open and transparent organization (an often stated goal with most organizations moving to agile), lies in the ability to share and visualize information across the organization. This must be closely couples to the objectives of a digital, agile and DevOps transformation.
Global Transformation Visibility Rationale
As an organization moves to the next levels of delivering upon the promises of digital transformation, driving ever greater speed and quality concurrently, the protection required through policy enforcement will allow that to occur in a safe and protected manner.
The key to ensuring the success and adoption of these policies over time will lie in the transparency and visibility of those same policies. It is for that reason that a dashboard (note: this is not a maturity model), will be required in order to adopt this visibility across the organization in a consistent manner.
Once policies have been effectively defined and established, visibility to showcase these policies, including the state of enforcement, will drive the next level of digital transformation. It will do so by ensuring transparency of the manner by which we safeguard our teams and organization as we pursue ever-increasing velocity and quality in our delivery. (Global Transformation Visibility)
Global Transformation Visibility Value
By ensuring that the definition and enforcement of an organization´s global policies are established in an open, visible manner, we understand that all members of the organization, including leadership and our development teams are driving toward the common objective to protect key behaviors while concurrently continuing to drive improved velocity and quality.
Once basic policy visibility has been established, building a policy visibility framework, inclusive of an educational progression model, is what will bring the behaviors to life and into the culture of an organization. (Global Transformation Visibility)
Challenge. Creating a policy visibility model is not without its complexity. Defining policies that are critical to the organization is complex, but defining the order in which these must be adopted, and visualized adds a new level of intricacy. That is, defining key behaviors, and ordering the importance of those behaviors on a global scale, is an iterative, evolutionary process.
Driving Change: One Common, Digital Vision
As we discuss each of the components above, there remains one open question related to this high level discussion. This surround the manner by which we will drive this change.
In a previous post, Digital and DevOps Transformation: The First Three Areas of Focus, I discussed the dynamic of establishing a catalyst team as one of the early, required drivers to digital transformation.
This team, made up of shared group of global and local (business, geography) team members, is responsible to drive the overall vision of transformation (in this, within the context of the DevOps ecosystem), and at the same time ensure local implementation within each of of the business units and/or geographic areas.
Global “catalyst” teams drive the high level strategy, and form part of one cohesive team with mirror teams in each business unit or geography, responsible to drive local change through ongoing pilot implementation of the DevOps ecosystem.
Foundation vs Next Generation Capabilities
This post focuses on “foundational capabilities”. In doing this, some obvious questions will arise surrounding additional capabilities required within a mature DevOps ecosystem. Some of these may include the following:
- Advanced testing capabilities
- Advanced user experience enforcement suites
- Component containerization and imaging enforcement
- Component third party component integration scan enforcement
To ensure clarity on the topic, the above points (in addition to many others, not mentioned), remain critical within a mature, global DevOps ecosystem. However, the importance within this post is to ensure that we focus fist upon building an adequate foundation, upon which all of these capabilities may later be placed.
While it is true that a very complex set of capabilities is required in order to form a mature DevOps ecosystem, organizations must first establish the foundation DevOps ecosystem upon which these more advanced capabilities will rest.
Conclusion
As we think about establishing a mature DevOps ecosystem, it is paramount to ensure the importance of building the appropriate foundation, upon which a more mature technology organization and ecosystem may rest.
In establishing this foundation DevOps ecosystem, below are several of the early components to consider.
- One Single, Global Process
- Global Planning and Definition
- Global Communities
- Common, Global Repository
- Global CI/CD Pipeline Pattern
- Global Delivery Visibility
- Global Product Orchestration
- Global Policy Enforcement
- Global Transformation Visibility
- Establish early “catalyst” global and local teams
Upon establishing these components, an organization may then begin to drive this DevOps ecosystem to the next level, including many advanced components required of a mature delivery and deployment organization.
The above depicts a perspective regarding establishing a foundational global technology ecosystem, and driving that culturally into the fabric of the organization. A cautionary note lies in that this is not a recipe, it is a trajectory. No two organizations are exactly the same, and the manner in which these components are achieved must always be calibrated locally. However, the components required to drive these change across organization have a great deal of commonality across many large scale transformations, organizations, and industry sectors.
References in this article:
Scaling DevOps: The First Three Areas of Focus
