DevOps transformations begin with a mindset change
I have spent the last eight years working with the developers at my company setting up and improving our build and deploy processes. In that same time, I have worked with dozens of Development and Operations teams to assist them with their automation capabilities. My Operations team has expanded our capabilities from code delivery pipeline to infrastructure as code and containers. I have also spent the last year presenting on DevOps topics to a handful of regional conferences in the Midwest.
In every conference presentation about DevOps and in every first time meeting with a client team, not one developer or client has asked me how to get started with a “Digital Transformation” or a “DevOps Journey.” Instead they ask, “Do you use Jenkins” or “Are you using containers right now?” (the answer from me is a resounding “yes”). Automation and continuous integration methods undoubtedly help development and operations teams work together and increase delivery in that part of a business. Yet these tools do not answer how to bring about the cultural change necessary to adopt a new philosophical ideology. You can do more with DevOps by looking at the flow of the work through the entirety of the system. At its best, DevOps enables the business to create less risky estimates, helps QA and Security teams save time with automated tests, and solves the problems of users, but the DevOps mindset requires more than the ability to implement a set of build tools.
Many companies assert “we can’t do DevOps here” for a litany of reasons: organizational complexity, team structures, lack of buy-in. But the start of a DevOps change only requires an individual to reach outside of their own team and processes. A singular decision to consider other people can lead to positive changes affecting great swaths of an organization in days and weeks instead of years. While the mindset changes straightforward and simple, they are difficult for most people to start doing. A DevOps journey is never really complete, but two things can catapult you to winning over teams and buying back the time you need to start building out the tools and systems you need to succeed.
Start with empathy
The first and most important thing you can do is to empathize with people. I would be wrapped up in the “things I needed to do”, so I was too busy to help others. I said things like, “Sorry, I can’t help you, I have to take this ticket and go run this command,” or “The team is probably going to ask for a deploy in the next two hours so I better not start your interesting problem or it will get interrupted.” There was also a lot of “This server is basically on fire, so I have to go put it out.”
My team was part of a software factory pipeline where work came to us from another “station”. We did something and then passed it on to another group. The business moved tickets from Development, to Operations, to QA for each development, staging, and production site. A delay in any of the ten steps in this pipeline led to angry clients. And Operations was often the cause of the delay to actually “see” the changes in production. We did not know the development teams were annoyed at relying on us to deliver through that factory line. We also did not know that the business had promised a certain level of delivery speed that we could not meet, because we spent 30% of our time putting out fires. And we did not realize our own Operations team was at its limit for pain and frustration. It took stepping out of our regular cadence and talking to the people around us to understand what the problem was.
Hero mentalities led to many overwhelmed hours and sleepless nights. It took me about 3 years in my current job to realize that helping others solve their problems was beneficial to me in the long run. We had a team member quit because they were bored and burned out. Another Operations team member and I took a “no deploy day” to talk to developers and the project managers to figure out how to replace someone doing fifty hours a week of useful work. Solving the problem included writing a ten line bash script, tying it to Jenkins, and setting up fifty jobs that the teams could run whenever they wanted. A team of ten people that previously received more than thirty weekly off-hours deploy requests now received five or fewer per week. The clients were seeing release-ready code get to production in weeks instead of months, which made the business happy, too.
Optimize the big picture
The next thing to do is look at the big picture. Understanding the holistic pain from three different organizational units pointed to a problem with the workflow. We recognized the issue and created a technical solution to solve that problem. Bash commands run by a Jenkins job freed up 65% of the time of six people.
Nature abhors a vacuum, and in business, work fills that void. After showing these amazing results, we were able to take on clients of increasing size and complexity. In one particular project, our deployment pipeline solved only half of the problems the teams were facing with delivery speed. We collaborated with the development teams to understand the problem of onboarding new developer resources. We realized that while we were pretty good at setting up systems quickly, our new developers were taking weeks to install and test applications locally. We needed to replicate our shared development system success all the way down to the developer laptops.
It took a couple months, but centralized the source of application binaries and code, and used Vagrant to create system images. A developer could download and run a command to install the application and code from these central sources of authority. By front-loading some time with Developers and Operations teams working to create downloadable images, forty hours of effort was reduced to four hours to get a new dev system up and running with a local instance of the application and code to test.
It was an obvious solution that only occurred to us when we focused on the problems that people were having. Operations wanted to sleep at night. Developers wanted faster feedback on the results of their work. The business wanted fast turnaround of features on time and (mostly) under budget. Looking at the whole process to see what could be fixed to the benefit of all teams allowed us to find a win for all parties.
As we kept “buying back our time” to collaboratively solve peoples’ problems, we got as much leeway as we needed to get the job done. We rolled out persistent chat rooms, wiki pages for documentation, binary repositories, and worked with QA and Security teams to implement automated testing solutions. Over the years, my team has a long list of wins and solutions we have created by putting the people around us first. Understanding the pain others feel and solving those issues has led to a large amount of time saved and respect gained. In going from a cost center to a value-add, Operations is now much more likely to be included in the business development pipeline.
The methods DevOps brings to the table are not new to technology, but looking at the problems we faced in a new way helped reclaim my work/life balance and made my job so much more interesting. I am no longer called 24x7 to simply push a button. One day a week, I talk with teams and figure out what they need to do their jobs better. We look at tools and solutions like Hygieia dashboards for product teams, TensorFlow for data service, SonarQube for code analysis, and ZAP for security. All these intriguing problems are being solved now because I changed my mindset about how I worked with those around me. My team jumped on board, then another team, then another. Simple, but difficult. Empathize and understand people first, then look at the big picture. The journey is just starting.
