DevOps Automation
What it is, where it falls short and how we can fix it
“Success in software development requires that organizations empower developers, enabling them to build productively, collaborate globally & securely, and scale innovation.” -Microsoft and Sogeti Enterprise DevOps Report
I. What Is DevOps?
If you are a software developer or if you work with software developers, chances are you have at least heard of DevOps. But, what does it really mean? Gene Kim, software engineering researcher, author and former CTO of Tripwire describes DevOps as follows.
[DevOps is] the architecture, technical practices, and cultural norms that enable us to… increase our ability to deliver applications and services... quickly and safely, which enables rapid experimentation and innovation, and the fastest delivery of value to our customers… while ensuring world-class security, reliability, and stability… so that we can win in the marketplace. -G. Kim
That is a somewhat abstract definition. However, if we take Kim’s definition and distill that down to the origin of the word DevOps from Patrick Debois, which was a reference to collaboration between Dev and Ops, we arrive at the following, [slightly] more concrete definition.
[DevOps is] a multi-disciplinary approach to quickly and iteratively deliver value-adding software systems; DevOps is predicated on the unification of software development (Dev) and IT operations (Ops). -Blu Flame Technologies
II. The Components of DevOps Automation
DevOps is a cultural shift away from the siloed teams of yore, who begrudgingly tossed slices of work back and forth over a wall of process and paperwork. Certainly, the biggest component of DevOps is the cultural shift away from siloed functions and incentives and towards collaboration across job functions in the interest of delivering a complete working system. However, in order to effectively do that, some technology scaffolding is needed.
IaC
Infrastructure as Code (IaC) allows infrastructure to be treated a lot like software. With IaC, infrastructure (servers, networking, storage, services, etc.) can be defined in code. What’s more is that provides a repeatable mechanism for defining infrastructure and its lifecycle that can be versioned just like software… because it kind of is software.
CI
Continuous Integration (CI) has almost become synonymous with software automation. That makes sense since a significant component of CI is build automation. But CI, like DevOps is also predicated on a cultural shift — a shift toward frequent (≥ 1/day) code merges into a common code branch (integration). Each integration then has an automated build executed against it that validates the build and provides fast feedback so that breaking “integrations” can be detected and resolved quickly.
CD
Continuous Delivery (CD) is the regular and automatic deployment of software following a successful build. The implication is that there is some sort of delivery pipeline, where each stage provides increased confidence in the working system. This is where IaC really shows its value. Software automated CD can provision infrastructure using IaC and deploy software onto that infrastructure. That’s a little harder to do if you are handcrafting servers.
Continuous Delivery falls just short of automating deployments all the way through to Production. Instead, its goal is to ensure that the software system, while under development, is always in a deployable state. This may include human workflow gates. Removing such gates and deploying without interaction is Continuous Deployment.
Self-Service
While CI and CD are all about what happens after the code is integrated, self-service is more about empowering the people who are developing the software. To borrow from the Microsoft & Sogeti quote from above “Success in software development requires that organizations empower developers.”
Self-service takes many forms. It’s providing developers and environment where they can test and experiment. It’s giving them the tools they need. It’s giving them the ability to both build the software and provision the infrastructure. It’s empowering them to create a developer workspace that maximizes their productivity. Finally, it’s about getting out of the way and letting them create working software.
III. The Current State of DevOps Automation
It’s better than it was, but it’s not as good as it could be.
First, there was build automation with Cruise Control. Next, there was Jenkins and Hudson. And now, over 10 years after Jenkins’ arrival, cloud computing has become mainstream, and with it, infrastructure as code (IaC). And, then there is Docker & Kubernetes along with a plethora of niche tooling. But, what is the current state for most people and organizations? The answer to that question can best be summarized as “It’s better than it was, but it’s not as good as it could be.”
What’s Better
- Build Automation
Build automation has wide adoption, the cultural challenges of CI notwithstanding. It’s a fixture from large organizations down to small open source projects.
- DVCS
Distributed Version Control Systems (DVCS) with Git have taken center stage, providing greater flexibility and performance than its predecessors. Git has become also nearly ubiquitous.
- CD, Cloud Computing, Docker & Kubernetes
Cloud computing, Docker, Kubernetes and the rise of IaC has made CD, and a whole system approach to developing software systems more palatable. Docker and cloud PaaS offerings (including Kubernetes PaaS) have streamlined infrastructure development and loosened dependencies on enterprise application servers, giving ground to microservices architectures.
What‘s Lacking
- Build Complexity
Software has become more complex and, as a consequence, software builds have gotten more complex. Not to be left out, build automation has also followed suit. It’s not uncommon to work across multiple programming languages and tools in a single software project (e.g. Terraform, NodeJS, Docker/Kubernetes, Python). Similarly, it’s also not uncommon to have developers spending a significant portion of their time supporting build automation (writing build automation code, standing up build servers, troubleshooting, etc.)
- Project Onboarding
The infamous sprint-0 (aka the getting your s*** together sprint) has become nearly synonymous with setting up prerequisite tooling. Bespoke tools, custom configurations, heterogeneous software stacks and complicated build automation platforms all contribute to long project onboarding times. It doesn’t have to be that way. But it’s a painful reality for many teams far too often.
- CD
CD is not a reality for many, especially for those who are shunning PaaS offerings, like Kubernetes. But, even for those who have embraced PaaS, CD is still really hard. Why? Because orchestrating IaC and Web/General Purpose (GP) software is hard. And, most of the available tooling doesn’t handle both well. Sure, you can shoehorn CD into a CI tool. But should you?
- Self-Service
The single biggest shortcoming in the current state of DevOps is self-service. Undoubtedly, some of the fault lies with organizational politics. But, available technology solutions are also to blame. With rare exception (and no doubt some custom tooling), complete self-service solutions still elude software development teams. These are the very people who, according to Microsoft and others, need to be empowered as requirement of overall software development success.
IV. Fixing The Problems With DevOps Automation
Low-Configuration Project Onboarding
The faster projects can be bootstrapped with automation, the more time teams can spend building value-adding software and the less upfront (and ongoing) overhead they incur.
For fast project automation onboarding, nothing beats a simple low-configuration solution that handily works with heterogeneous software stacks. Low configuration being a key ingredient for several reasons: little vendor/tool-specific automation code/configuration to maintain, less drift (more consistency) between projects, and less effort (and therefore, time) required to create configurations.
CI & CD
The line between software and infrastructure is blurrier than it has ever been. This is in no small part because of public cloud offerings, IaC, Docker and PaaS offerings, like Kubernetes. But, there still exist very different and specific considerations for both build and deploy capabilities.
However, for the sake of DevOps automation, Dev (building software systems) must work together with Ops (deploying & monitoring software systems). After all, it’s DevOps, not Dev and then Ops. A CI & CD solution, must therefore, natively build software, deploy software through a deployment pipeline, and get feedback at every step along the way, including after deployment to production.
Self-Service
When considering what self-service is needed, it’s important to consider the ways in which developers can be blocked from completing their work. From that context, developers could benefit from being able to configure their workstations themselves, build software themselves, source dependencies themselves, provision production-like infrastructure themselves, and package & deploy software to that infrastructure themselves. The most common friction points are workstation setup, infrastructure provisioning and software deployment.
It’s a safe bet that if there are glaring friction points for developers in an organization, it’s rooted in one of those 3 aforementioned things. So, when it comes to the question of what should be targeted for self service, the answer is simple: empower developers to (1) setup their own workstations (2) provision their own production-like infrastructure and (3) package & deploy software to that infrastructure. Bonus points for making these self-service targets “one-button” operations.
Summary & Conclusions
The state of DevOps automation has come a long way since the term was first coined in 2009. Build automation, the public cloud and Infrastructure as Code (IaC) have become commonplace in many organizations. Although these technologies enable Continuous Integration (CI) and Continuous Delivery (CD), their adoption often lags technical capabilities. This is especially so for CD, due in part to operational complexities and organizational barriers.
However, DevOps automation still fails to address common friction points in some key areas. In some cases, automation software requires a substantial custom code and/or configuration investment, which can slow adoption and result in significant maintenance overhead. A low-configuration option can alleviate these burdens. Additionally, CI & CD capabilities are rarely well-integrated, with solutions tending to favor CI or CD. Effective CI & CD should be an integrated solution that natively builds and deploys software through a configurable pipeline.
Finally, self-service continues to be a persistent pain point, overall. From setting up developer environments to enabling developers to provision and deploy to production-like infrastructure… with rare exception, self-service is just not a reality for software development teams in many organizations. But, if success in software development requires that organizations empower developers, then it needs to be.
Thank you for reading. If you liked this post, please check out the Blu Flame Technologies Blog.
Resources
- Enterprise DevOps Report by Microsoft & Sogeti
- The Unicorn Project And The Five Ideals
- The Origins of DevOps: What’s in a Name?
- Developer Velocity: How software excellence fuels business performance
- Continuous integration vs delivery vs deployment
- Infrastructure as Code
- What is Kubernetes?
- State of DevOps by Puppet
- DevOps Culture: Self-Service-as-a-Service
