Three things your DevOps teams should embrace
How to deliver a steady flow of quality with your DevOps team
Have you ever visualized the technology value stream as a conveyor belt on a plant floor?
Looking at every feature or fix of code rolling as a small packages through the different plant workstations. Continuously deliver small packages of value from a one-line requirement phrase, to deployed value in production.
I never had this visualization in my head before until I read an unusual book. Which is one of my best reads lately: The Phoenix Project ¹. My biggest take-aways from reading the book was:
- We have a lot to learn from the Lean movement, powerful complement to agile and DevOps practices
- Continuously deliver tiny changes is always better than massive changes
- DevOps is much more than team formation and has unfortunately gotten an excluding name
- Identify, protect and eliminate your constraints, they limiting your pace
- Leaving space for idle time is essential
The Phoenix Project is a tech book written as a novel. About Bill Palmer that got promoted to VP over IT operations at his company Parts Unlimited. Following his journey from real chaos and endless fire fighting to starting to get some value delivered to the business.
The book gives a tremendously good example of an organization in the worst possible situation, stuck in long term projects, not seeing daylight. Revenue going down. Doing too many things at the same time and not delivering any value and struggling with heaps of audit issues. Also, worst of all — spending too much time on unplanned work in an endless downward spiral. Since the technical debts are not paid down and are eating up the time and resources, bit by bit, while competitors are running fast and winning new markets shares.
It’s a fictional story about a dysfunctional company on the thin line of survival, especially by all the recent IT screw-ups. Despite the extreme situation, I think many can relate to some of the difficulties and frustration they are facing during software development.
The three ways
The book gives a lot of real-world pedagogical examples of how powerful the DevOps principles can be. The authors are referencing the “The three ways” as things Bill and his coworker need to master to get out of the miserable situation.
- The flow — Speed up the flow in the technology value stream. Create a fast and smooth flow from left to right.
- The feedback loop — Create a fast feedback loop to ensure quality and reliability. Release often and small. Simplify and truly automate the deployment pipeline and environment setup. Use monitoring tools to identify issues quickly.
- High trust culture — Create a culture that supports experimenting and understands that fast feedback from failures is the best input to progression. With risk calculated and manageable by the fast and small automated deploys.
“Create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery.”
The flow
My takeaways from reading this book are many. However, regarding the flow, this is the ones I prior highest:
- Reduce “Work In Progress” — It’s hard and demands courage and discipline to freeze ongoing activities, and it can be hard to realize doing less will speed up the throughput.
- Identify your constraints/bottleneck Try to find your weakest link in the development and release pipeline. For example: in the book, everyone is dependent on a guy called Brent. Almost every activity needs some work from Brent before it can proceed. He has through the years acquired knowledge no one else has, and everything is in his head with no documentation. Which results in, he is steering the overall progress and velocity. Is some resource/team/developer needed by many and holds up overall progress in your organization?
- Protect and eliminate your constraint — When you have found the constraint. Make sure it is as efficient as possible, if possible, protect your constraint from distractions and unplanned work. Try to find a way to eliminate personal dependencies causing the constraint. Try to spread competence and minimize dependencies.
- Visualize WIP — Make it easy to see whats is in progress (on Kanban boards for example). Also be strict about showing what is in progress, planned or unplanned. The overview gives control, and it’s easy to see how new priorities affect the output delivered to the business.
- Integrate compliance and security as a value-adding part in the development process. In a way, it prevents the business from potentially unnecessarily risks ahead of time.
- People and teams needs idle time. I had heard this before and didn’t care so much about it. However, it blew my mind how important it is from reading this book. Especially if you have internal dependencies between individuals and teams, the authors are using the formula busy time / idle time = wait time. For example, if one developer is 95% busy and idling 5%, the formula is 95/5=19 hours to wait before releasing resources for others. Of course, the formula is theoretical and works by the assumption that people aren’t context-switching. However, I still think this formula is on to something. On a larger scale, it can almost end up with a distributed deadlock. Everyone waiting on each other to release time for you, but they can’t because they also are stuck waiting on others like an endless loop of lead times.
My reflections
We live in a complex and ever-changing world. We need tools to detect and respond to changes.
When implementing DevOps principles, I think it’s essential to have a broader perspective and understanding as this book gives a good example with The three ways. The book often compares the development progress with the progress on a plant floor. It pushes you to get a visual understanding of the flow from raw inventory materials to a working product. We have a lot to learn from decades of optimization and research in the plant pipelines. We may have more in common with the lean movement than we think.
More than just team formation
Previously, my naive understanding was that DevOps was only about team formation. Bring developers and operations people together and give them responsibility for the environments and applications running, and then you’re done. (With the friendly addition of joining 24/7 on-call lists when things start burning. Calls you want put down when sitting on the beach with your kids Saturday evening)
The authors behind The Phoenix Project have a more significant take on implementing the DevOps principles starting by The First way focusing on getting the flow working as a first hygiene factor. Create a proper flow to control what’s going on. Then use the feedback from the second way as input to get rid of technical debt causing unexpected outages. Lastly, The third way gives freedom and trust to the team to experiment and test new things. Space to fail and learn and most important of all to have fun at work.
“Business agility is not only about raw speed. It’s about how good you are at detecting and responding to changes in the market and being able to take larger and more calculated risks.”
The team formation is, of course, crucial. Many are already doing it in cross-functional agile teams. The book stresses the fact and importance of having developer skills, operations skills, QA skills, security skills, and business skills, nearly working together in the same team and iteration.
Involvement is important
My experience is that the higher the level of participation and understanding of the business I have as a developer, the easier it is to create the right technical solutions. Also, even more important is that the motivation and excitement tends to go up in the team.
As a developer, I want to understand how our organization is creating value and profits. Likewise, I’m convinced it’s equally essential for business people to understand how we tech-people think and solve problems. I have experienced it multiple times in real life and believe it’s a win-win situation.
Conclusion
A lot is about automation, simplify flow, and make it all visual to others to follow and understand. Ship often and small. Fail and learn from it. Repetition and practice leads to mastery.
My conclusion is that an allowing culture is the single most vital ingredient to make it all work. An open and positive climate that allows failures and always supports new experiments and focusing on learning. The high trust culture is an essential must to evolve and succeed when embracing the DevOps practices.
Not all walk the extra mile and put in the effort to achieve multiple daily deploys. However, I think it can be a game-changer. It pushes the culture and infrastructure of continuous deployment to a new level.
My first medium article based on my reflections after reading the book:
[1] The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win, written by Gene Kim, Kevin Behr, and George Spafford. 5th Anniversary Edition from 2018.
