I should have your attention now. I can only imagine that you, dear reader, currently hold the title of "DevOps Engineer" or have a few colleagues that share this job title, and they, I am sure, are not imaginary. You could be a manager hiring DevOps Engineers right now and might wonder if this poor author may have just lost his marbles. You might be somewhat correct. Railing against a pervasive industry trend could either mean I'm on to something or… we know the alternative.
It is not the effort of such engineers or the quality of the work I have a problem with. By most accounts, the work I see in the wild is good and fulfils the intended purposes well. What I have a problem with is the title itself. "DevOps Engineer" and the purpose for which they ardently spend their engineering effort.
So, what could go wrong with handing the title "DevOps Engineer" to several (probably very hard-working and highly skilled) engineers? Well, nothing is going to go wrong per se. There will undoubtedly be benefits to the output of your development teams (both qualitative and quantitative). And you and your team will work along your merry way, doing the DevOps… or so you think.
The problem I have with the title "DevOps Engineer" is the general perception of many enterprise leaders is that now we have a DevOps team full of DevOps engineers, we must be doing DevOps, right?
Umm, No.
If you have DevOps engineers toiling away in a DevOps team. You are not doing DevOps.
So what is DevOps?
There are plenty of explanations on the web of what DevOps is. If we are to distil its meaning to as few words as possible, we will likely end up with something like this:
DevOps is a set of practices, guidelines, and culture designed to break down silos in IT development, operations, networking, and security
Ok, so I may have just pulled that from Google's definition. But they are arguably one of the most successful software companies. If you were to ever work at Google, you would know that they don't have DevOps engineers. (I'm aware they have a DevOps Engineer certificate for GCP. Unfortunately, this mainly panders to enterprises who insist on such practices.)
I also like the following from the Phoenix Project:
The application of Lean practices and principles to the IT value/work streams.
If I were to simplify this a little further to its most core component, though, I’d say: “DevOps is a process”.
This above flow diagram is ubiquitous to DevOps. How do you know you are starting to do DevOps correctly?
Your teams go through this process multiple times a day, culminating in multiple deployments to production in a day.
Why is the DevOps Engineer title wrong?
The role title leads to an incorrect perception of what DevOps is and isn’t. And what DevOps isn’t is an engineering role.
Let me explain:
Thousands of businesses out in the wild believe they do DevOps because they have DevOps Engineers. Simple as that.
They believe this because they have a score of DevOps engineers forming a DevOps team that has built a Jenkins sitting in the corner of their cloud platform running all their code pipelines. Yet each release still takes weeks to plan, weeks to test and deploy.
Yet operations teams still struggle with midnight releases and arduous user acceptance testing. They still end up with mostly unreliable software, which leads to the awful affliction of “operator crying in the corner of the server room syndrome”.
This is not DevOps.
How do we hire for DevOps, then?
If your developer and operations teams are of a decent engineering calibre. You shouldn't need to hire anyone new to achieve DevOps nirvana.
All you need to do to start your path to DevOps is give your developer teams and your operations team the same goals and let them work on this together. Leadership and trust are the first stages of DevOps (in my admittedly ample experience).
It helps here if your operations team has a member or two who have a black belt in their favourite programming language (Python, Ruby, Go, Rust, TypeScript etc.) and a software engineer who certainly knows a thing or five about Linux, networking, systems engineering etc.
What else do we need to do?
Leadership:
Organisations will only change with the right advocacy and support from management. In this approach, it is of the utmost to make sure that the business goals are clear and that every engineer in your technical team is well-equipped with the knowledge of what the business wants to achieve and how to attribute these to the goals they will be setting for themselves. Ideally, leadership should be familiar with LEAN principles.
Introduce Tooling:
The goals set by the engineers should inform the next steps and which tools they will need to add to their tool belt. A lot of teams may jump for the CI/CD automation first. I’d suggest avoiding this. Looking for appropriate monitoring, logging, alerting, and dashboarding tools first is a great way to avoid early pitfalls. If your product lives in a public cloud, using their in-house solution is a relatively low-cost and efficient way to get started here. They also come with the bonus of being able to drive further automation. On that automation note: keep this in mind. The tools you choose should be able to work together, and don’t ever choose tools that don’t offer an API. Following monitoring, look at various ways you can test your product’s code. Test the most problematic code first; start with parts of your code that cause the most problems. Go from there.
Patience:
Adopting DevOps takes time. The bigger your technical team and the scope of work they deal with, the more technical debt they face will factor into how long this will take. A good incremental approach will take a few years for any medium size business with a fairly large technical team (say 50 + engineers). Expect that this will, over time, never be finished. You will always find ways to improve.
Trust the process:
I know many managers, executives and non-technical folk, may feel the need that they must interject in the overall progress. Planning too much too far ahead goes against the benefits that DevOps (even agile) can provide. Your goals as management, in this case, are to set goals, make sure everyone has everything they need to work at their best, provide knowledge (technical and non-technical) and be a great interface for the business.
Avoid performance management by numbers; instead, focus on outcomes provided to your customers and their experience and satisfaction interacting with your product.
Stop Aiming for Perfect:
DevOps, just like Agile, is about good enough and continuous improvement. When starting in DevOps, you may find yourself aiming for perfection. Instead, start minimal, start small, and start simple. Picking low-hanging fruit will give you a nice early lead and help free up time for the harder stuff in the future.
For operations teams, do not aim for 100% uptime. This is completely impractical and will only serve again to work against you. Pick a sensible uptime target with management, considering all other systems your product depends on.
Monitor your error rates. This will feed into and give you indicators of your uptime. Google has a fantastic section on this in their excellent SRE handbook here.
Other notable challenges and things to consider
As with most things, DevOps will look a little different from team to team, business to business, depending on your industry, regulation, product types, country etc.
Regulations:
Some industries have significant regulatory frameworks they must adhere to. While some regulations can pose some difficulties for teams wishing to implement DevOps, these are very rarely insurmountable and require some out-of-the-box thinking. I have made a bit of a name for myself in the past surprising regulators with solutions allowing a DevOps workflow that comply beautifully with regulation.
Team topology:
DevOps can look a little different for each business that looks to implement it. Typically how your teams interact, the types of teams and the makeup of each team can vary depending on your business's needs. Some classic anti-patterns should be avoided. Included the all too common “DevOps Team with “DevOps Engineers” anti-patterns.
What do we hire when we need more people:
Hire for the same roles you were hiring before, but consider how work is approached in the DevOps way; A team may become a little more multi-faceted. An unfamiliar new role may enter the arena over time, though. The “site reliability engineer”. They are essentially a software engineer trained in building tools to assist with building a reliable customer experience with your software products.
By all means, though, keep hiring systems engineers, software engineers, data engineers, etc.
My final recommendation, the more people that can code in a language, the better. I can not underscore this enough. A systems engineer, network engineer, data engineer or any engineer who can’t or will not code eventually becomes a liability in a DevOps environment.
Conclusions
I hope this gives some insight into your upcoming journey into DevOps. Or I hope it has highlighted some improvement in how DevOps is being pursued in your teams now.
Remember, DevOps is a way of working. It is a process, a culture shift that requires working more closely and collaborating more with other teams. Management needs to set business goals, attribute technical goals, and ensure that any regulatory requirements are covered in your DevOps process. Then, they need to trust the process and their engineers.
Finally, DevOps is NOT an engineering role. There is no such thing as a DevOps engineer, and a DevOps team will only work against the efforts covered by your engineers on their DevOps journey.
Errata and Modifications
- You don’t apply a process to a team. Your teams perform their work to a process.
- I added a definition of DevOps from Phoenix Project.
