Don't Let Planned Work Ignite With Incandescent Fury — Takeaways from "The Phoenix Project"
Reading 25 books in one year, that’s the challenge for 2018. Whether it’s fiction or non-fiction is not important, as long as it’s a book that can be read. You can follow my progress on my personal Goodreads profile.
Reading a book that explains the principles of DevOps can quickly become boring. If the only thing the writer provides you with are dry facts, it's hard to remain focused on the surely interesting theories that are described.
It's already better if the writer gives a lot of examples from the real world, because that can help you to better relate the stuff you read to your everyday world.
But it's actually the best if the writer (or in this case, three writers) decide to use storytelling to bring you the key concepts of DevOps. That is exactly what Gene Kim, Kevin Behr and George Spafford did by writing The Phoenix Project, subtitled "a novel about IT, DevOps, and helping your business win". OK, that last part is a bit meh, but don't let that influence your thoughts on this book. In The Phoenix Project we follow Bill, who is IT manager at Parts Unlimited. He's the man who becomes in charge of delivering the over budget and very late Phoenix project. He does so with the help of his team and the mysterious Erik, who introduces him into The Three Ways and other things to help the project succeed.
As I'm very interested in learning things through storytelling, I couldn't wait to start reading. Now that I have read it, here are my top five takeaways from The Phoenix Project.
The Three Ways of DevOps
It seems to me that if anyone is managing IT without talking about the Three Ways, they are managing IT on dangerously faulty assumptions.
In The Phoenix Project, the foundation that all DevOps patterns build upon are The Three Ways. It is a set of three principles that "describe the values and philosophies that guide DevOps processes and practices". Yes, The Three Ways do actually sound like some holy, spiritual concept, but chances are that you're already using these principles on your current job. That's no wonder, because methodologies like Agile and Scrum are heavily based on these "Founding Fathers" of DevOps. In short, these are The Three Ways:
- The First Way: left-to-right flow of work from development to IT operations to the customer.
- The Second Way: constant flow of fast feedback from right-to-left at all stages of the value stream.
- The Third Way: creating a culture that fosters two things: continual experimentation and understanding that repetition and practice is the prerequisite to mastery.
It's cool to see that a methodology like Scrum (or Agile in general) indeed follows these Three Ways perfectly. The right-to-left flow of work from the First Way is done by attaching a state to a user story and its tasks, where states could for example go from New, to Active, to Review and finally to Done (or Closed). The constant flow of fast feedback from the Second Way is a vital part of Scrum, where short cycles and continuous feedback from the end user are one of the key principles. And also the Third Way — a culture of continual experimentation and the concept of "practice makes perfect" — can be mapped onto Scrum. Fail fast and learn from the mistakes you make in order to be better in the next iteration (which can be discussed among team members during a retrospective).
More about The Three Ways can be read in a blogpost that was written by Gene Kim, one of the authors of The Phoenix Project.
Four categories of work
Like matter and antimatter, in the presence of unplanned work, all planned work ignites with incandescent fury, incinerating everything around it.
When thinking about your day-to-day work, the majority will probably consist of project work, like adding new features to an already existing project or writing a new software program from scratch. But next to this, there are actually three more types of work that you'll encounter. Bill, the main protagonist of the book, is introduced to these four categories of work by Erik, an Agile evangelist who is the driving force behind the resurrection of the failing Phoenix project.
The four categories of work are:
- Business projects, initiated by the business itself.
- Internal IT projects, like setting up new environments, automated deployment, etc.
- Changes, often generated from the previous two types of work, like fixing feature defects (bugs).
- Unplanned/recovery work, like operational incidents and problems which come at the expense of the previous three types of work.
The first type of work should be the main focus within an organization, but often the other types of work (especially the last two) are necessary because of a lack of quality in this first type of work. For example, work that falls into the category of Changes is necessary because the introduction of bugs in the first category of work. The same is true for unplanned work like hotfixes, which — in theory — wouldn't have been necessary if the issue was noticed while working in the first category of work. That is of course the theory, because in practice you'll never be able to introduce zero bugs into a software program. But we're always striving to live in an ideal world where Changes and Unplanned Work are non-existent.
Bottlenecks and resource wait time
He pauses and then says emphatically, "Eliyahu M. Goldratt, who created the Theory of Constraints, showed us how any improvements made anywhere besides the bottleneck are an illusion."
The concept of "resource wait time" is a very powerful way of showing the devastating effects of work lying around in a queue somewhere, waiting to be picked up. The graph above shows that the busier a resource is, the longer the wait time becomes. Especially when a resource is busy for 80 percent of the time or more, the total wait time goes through the roof.
A resource that is busy for most of its time can quickly become the bottleneck in a workflow. In The Phoenix Project, this busy resource is actually a person called Brent, who has so much specific knowledge of the software that every piece of code must go through him before it is accepted and can be used in production. Another example is a thirty-minute task that requires seven handoffs before its ready to use, making the actual wait time count up to 63 hours (nine hours for every queue the task waits in at every handoff). Because of this enormous waiting time, the actual percentage of added value is only 0.16% (thirty minutes divided by 63 hours). This example shows that it's extremely important to make sure you have as little handoffs as possible, or at least try to make these handoffs take as little time as possible.
Technical debt and the IT capacity death spiral
If an organization doesn’t pay down its technical debt, every calorie in the organization can be spent just paying interest, in the form of unplanned work.
Technical debt, what a joy! It's not without reason that technical debt is called "the IT capacity death spiral" in The Phoenix Project. The more technical debt you have, the more time you need to work around issues that were introduced in the past. And this — of course — leads to even more hacky code in the future, until the entire software falls apart like a house of cards. It also relates to unplanned work, the fourth category of work described before. Not handling technical debt makes sure that you spend most of your time in unplanned work, purely because no one knows the impact of writing a new feature or changing some piece of code. It's a no-brainer to ensure that technical debt is dealt with as soon as possible. Some even say that you should use as much as 20 percent of your working time to resolve technical debt, and that's not a bad idea at all.
Rushing functionality and sharing knowledge
Every time that we let Brent fix something that none of us can replicate, Brent gets a little smarter, and the entire system get dumber.
All developers need their breaks, right? You can't code for 8 hours a day, 7 days a week, 365 days a year. Well, you can… for one or two years and then burn out. If that's what you're aiming for, great. For the rest of us, holidays are a way to charge our batteries. But what if you're the only one who has specific knowledge about a certain part of the software, and your holiday is approaching? Chances are that you're forced to rush your coding and make sure that all work is done before you take your well-earned break. It's not hard to predict that this can lead to huge problems.
The main thing here is that the specific knowledge you possess is not shared among multiple team members. It's vital to share knowledge of processes among team members, so that at least two (but preferable more) people know how to solve things if shit hits the fan. It's either that, or you're going to be called back from your holiday to check this issue only you know how to solve. And that's the last thing you want.
In The Phoenix Project, this "know-everything" person is — again — Brent. Throughout the book, Brent is classified as the bottleneck who possesses too much unshared knowledge. And also for the members who work on the Phoenix project, it's of the utmost importance that this knowledge is shared to avoid future catastrophes.
Did you find this article useful? Consider clapping for it! And if you're interested in finding out how the issues the developers of the Phoenix project faced were resolved (or when you're interested in DevOps in general and like to learn more about it through storytelling), definitely consider to buy The Phoenix Project from your local bookstore or the internet!
