State of DevOps on Windows and .NET

Disclaimer: I’m only a recent full-timer to DevOps “proper” (i.e. on a DevOps team tasked with implementation/advocacy). The opinions and views here might be a bit premature. That said, I still think they’re valuable given the vast majority of the industry is focused on DevOps right now, and that kind of effort needs to work for all kinds of environments and businesses. I’m also going to assert the open source world needs to take some responsibility here and live up to their own standards. Of course, I’m open to feedback.

Simply put, I love DevOps. I’m a 14 year veteran from the software side developing business applications. In terms of finding your passion, I would have to say DevOps is as close as I’ve come to it so far. It’s solving interesting business problems, actively seeing improvements in communication and efficiency in teams, being able to objectively prove your successes in a way that maps closely to business objectives, feeling the rush of commanding fleets of servers and apps, and a dash of spicy production support from time to time to keep you on your toes. That said, I’m still a bit letdown.

The industry standard here can be summed up in one word: Linux. You will be hard pressed to find robust tooling that works both natively and well on Windows with anywhere near the breadth of tools available to Linux environments. Yes, there are some Windows ports and many tools do work with or being hosted on Windows. However, the main core to all things DevOps — that is, pipeline automation and configuration management— still lives almost exclusively on Linux as neither Puppet nor Chef have native Windows support. To some degree, this should be expected.

DevOps tooling and modern deployment workflows largely got its start in Linux environments, and a lot of concepts originated from there. However, the trend is almost entirely one-sided and would stand to gain a lot more by a balanced distribution across the two main server environments.

I had written this blog out a few weeks ago but was initially uneasy about some of the assertions solely because of how new I am to the role proper. Then I read a recent thread on the /r/devops subreddit titled “My devops interview experience, are they all like this?” The author’s main question is if advanced Linux knowledge is required for even basic DevOps roles. Here are some of the responses.

If I may. Linux knowledge is preety[sic] basic and the whole cloud is built upon Linux. In my opinion it is must.

This is a fairly interesting and incorrect take. Azure, Microsoft’s cloud platform for those who might not know, definitely has Linux servers no doubt and has even incorporated it into its physical switches. However, make no mistake the majority of Azure is still Windows through and through.

Devops is BOTH Dev and Ops. You need skills on both sides of the equation. A Dev who doesnt understand the basic functionality of the systems they are building tools for is not going to do well.

I actually agree with this one and is why I say DevOps should be about supporting applications where they live. However, it’s a fair twist of logic to assume all DevOps should be Linux based because that is by extension saying all software is Linux based which is simply wrong.

Devops are supposed to be Dev and Ops. Config is pointless unless you understand what the config is doing.

Another comment implying all environments are inherently Linux based and therefore so is the software.

If you do not have a strong understanding of linux fundamentals you cannot be a capable infrastructure engineer working in a linux shop.

This is another interesting logical leap. The author was asking about “all” DevOps interviews. Somewhere along the line this commenter has associated “DevOps” with “Linux,” though I can give this person the benefit of the doubt as the question was about Linux. Still, this illustrates the strange relationship between “DevOps tools on Linux” and “a Linux shop.” The state of the tools as is does not describe the kind of shop a place is.

The above certainly isn’t a scientific study on the perceptions of DevOps fundamentals. However, it is a very interesting snapshot in time on how the thread author’s question is answered.

For those familiar with the history of Windows/Linux competition, this may feel like some comeuppance. The other shoe has definitely dropped, and the behemoth has found itself at a stark disadvantage. That said, it’s been long recognized that Linux has “won” the server space. Even so, Windows Server has a long history and very robust features that have been and will continue to be used en masse in tons of environments from small to “enterprise.” I would wager that if DevOps wants to shift into the next gear then the big name tools will have to open themselves to these large, sometimes arcane environments and even those new shops who consciously choose to utilize Windows Server and related tools even if they’re in the minority.

To be fair, Windows has classically been difficult to automate. However, PowerShell over the last several years has largely removed this difficulty. Further, there are other options such as Windows Containers and Windows Server Nano that make spinning up these environments fast. In adopting DevOps friendly practices, PowerShell even has Desired State Configuration that gives a declarative solution to administration. That said, I do recognize that most large tools can speak PowerShell or have Windows based agents that do, and that’s great! Unfortunately, the command servers for these tools are hosted on Linux, Linux expertise to setup and administrate, and Linux development experience if you want to extend core functionality. This is where I think the open source side industry has caught itself in some doublespeak.

While the DevOps.com 2018 State of DevOps Tools report has several tools that support Windows among its 13 categories, you can’t enact much change without Linux. You can communicate, organize, and build your changes all day long, but you can’t push the changes without Linux. You can not mutate your infrastructure if you take a tools based approach to deployments which you realistically should. You can not effectively “do” DevOps without Linux. This is a terrible bottleneck as the executor of change needs to be able to adapt to the apps it is handling.

Microsoft has done a lot to open its coding and tools to embrace an open source mindset and practice and being more visible and responsive to common feedback. This was done largely to compete with Linux as Windows was finding itself more and more behind. I can remember a lot of folks would crucify Microsoft for failing to keep up. Advance to today, I would assert Microsoft has lived up to its end of the proverbial bargain. It’s time for the open source world to recognize this and step up to support the opportunities these efforts have opened up.

Forcing environments to keep any kind of server, Linux or Windows, just for the sake of tooling support is honestly a bit rude and anachronistic. The landscape, and I would daresay spiritual goal, of DevOps should be to support environments and applications where they live and in a manner that is natural to them and their business practices. Right now, that can’t be done for essentially one-third to half of the environments out there by number, and I would wager closer to two-thirds to 75% of all environments by revenue.

One of the biggest stumbling blocks to any change in a software environment is being able to safely overcome old assumptions. You can’t hope to improve an old system even in objectively better ways if you can’t guarantee you won’t break existing flows. This is more and more true the larger and more deeply central the system in question is and doesn’t even have to be in terms of extending functionality. Being able to lift applications to virtualized hardware on its own can be a massive savings.

This brings me back to my comment above about DevOps shifting into its next gear. There are entire server fleets that are forced to work in incredibly inefficient and old ways because they can’t introduce unsafe change. This means you need to be able to meet them where they are. This means you need to be able to live peacefully next door to them, because they’re not going to drive a town over just because you have the shiny new store.

I’m not talking about the Netflixes or Amazons of the world who pioneered DevOps tooling as we know it today. Naturally, they are A-OK. I’m talking about the Chases, IBMs, AT&Ts, and Dells: these large entities with decades old environments that can vary wildly from each other. This isn’t to say those environments aren’t already utilizing the big name tools like Chef, Puppet, Ansible, and so on in really cool and really important ways. I’m saying there’s a largely untapped oceans in these behemoths that can’t be touched because the tooling currently refuses to leave its perch. To put it differently: think new money vs. old money.

As to how this affects me, personally, I work in a Windows environment where .NET drives our business applications. I have been in Windows/.NET environments for each of those 14 years including my current environment and where I was able to transition into DevOps proper. Bringing in a lot of tooling and processes that are advocated in various conferences and nascent “best practice” articles is impossible for me.

Someone could try to make the argument that the Windows platform is outdated or that non-core .NET is the Wrong Answer (TM), but they would objectively fail. Both Windows and full-framework .NET as platforms have very good, reasoned use cases. Teams and companies also have a velocity and life of their own. Teams also come with cost. This has put me in the unenviable situation of being on a small team with wide access to all levels of the org, a place where agility should be really high, where there is a lot of stagnancy in what we’re able to do out of the box with popular offerings.

To hire new talent takes money that isn’t in the budget. To cross train someone or send them to bootcamps takes productive time AND money the business can’t afford. To fire someone to make room in the budget loses knowledge and velocity that is extremely undesirable on the best of days. Then to square that up just so we can come up to speed where DevOps could potentially take us just over the question of operating system and tooling feels downright insulting.

There’s also the other aspect of looking for DevOps roles where you can’t effectively get hired if Linux and the big name tools aren’t predominant on your resume. Again, that’s fine for those environments whose business applications live in Linux. My point is this is a problem when the introduction of separate infrastructure is required only for the tooling. How can you reason effectively about the applications themselves if you require a different set of expertise to support them?

Even so, I’m still able to bring really cool things to my teams that people are excited about. What’s a letdown is that I am actively prevented from following best practices that I actually agree with. I would love to have open-ended tools that interact with Slack out of the box, for instance, that lend themselves to setup workflows with the systems I actually use and that my business relies upon. Unfortunately, I’ve spent the last couple weeks writing some custom tools because the big names don’t exist outside of Linux. My only real saving grace has been the systems I use happen to have APIs — thankfully, something that has more or less been a basic feature of popular systems.

All I’ve seen so far is the big names using their position to double down instead of open up even though Microsoft was crucified for the exact same behavior. For instance, the Chef “Windows” page would practically have you believing, as a Windows-based environment, that you can use it in your existing environment and largely be good to go. It’s even under a partners URL! Then you click on Downloads and find Chef Server but note a complete absence of Windows Server. Shit.

Right now the only third party tool that I know of that provides robust support for my environments is Octopus Deploy. Disclaimer: this isn’t an ad; they don’t know about this article at time of publishing; etc. etc. As great as this is, I don’t see any good competition to keep it sharp over time. It has a good community around which is a plus. VSTS, now Azure DevOps, has a CI\CD pipeline, yes, but is tied to a particular vendor.

This is why I say the Linux and open source worlds need to live up to their own standards and explore islands that aren’t their native homeland not simply to see the sights and to be able to say they’ve been there, but to make a presence and to get to know the inhabitants.