Lean Web Operations — Planning for the Unpredictable (I)

This year in October, our managed hosting platform freistilbox had its fifth birthday. I founded freistil IT in 2010, so five years is a major part of our whole journey. Unfortunately, in all honesty (and I’m going to be candid throughout this article), we did not come nearly as far as we should have at this time. We could have a better product, and we could have grown faster, both financially and in team size.

The reason for this lack of growth is not that in a small, bootstrapped company like ours, capacity is always tight. Other small businesses face the same constraints and manage to achieve more impressive growth. Blaming our lack of progress solely on not having enough resources would be a cop-out. Progress happens when you use the resources you have effectively. In this article series, which is based on my talk at DrupalCon Vienna 2017, I’m going to share what we’ve learned about making progress.

• Part 1: How we got ourselves into trouble
• Part 2: How we learned to keep work in flow
• Part 3: How we learned to make quick decisions
• Part 4: How we learned to manage interruptions
• Part 5: How we’re doing today

Who needs project management anyway?

I’ve asked myself this question often in recent years. My answer, every time: “Well, not us, that’s clear.”

First, project management seemed like something only bigger teams need. If you have to coordinate work in a bigger group or even across multiple teams, having appropriate processes and tools in place makes perfect sense. However, you can count the people in our company with two hands. Waving Gantt charts in front of such a small team seemed just silly.

Second, I hand-picked every single team member myself. They’re people I’m proud to have in my company, all professionals who love their work. I wouldn’t hire someone who has to be told what to do, would I? Over the years, I’ve learned to manage my own work. There’s no prominent productivity method I haven’t tried over the years. So I thought that, as long as everyone puts in a bit of time to self-organize, we’re good.

And after all, we’re working in an agile way. Experience taught us that planning too far ahead only sets us up for disappointment. In web operations, you get interrupted far too often for long-term plans to make sense. Instead, we decided to take it one task at a time. Reaching a destination is simply taking one step after another, isn’t it?

So many misconceptions.

The reality of web operations

What is it like, doing web operations? If you ask me in my sales role, the answer you’ll get will remind you of a pastry chef: The managed hosting infrastructure our engineers build is delicate and inspired, a bit like a soufflé. This is a job that takes well-honed skills, the most exquisite open source ingredients and surgical attention to detail. And the result is delicious.

However, if you ask our team, you’ll get the impression that web operations is like playing Dodgeball. How do we envy our web development customers for being allowed to focus on their work for hours, only getting interrupted maybe once a day by a project manager asking for a progress update! (Well yes, that is how we imagine it.) For us in the engine room, though, it’s a constant stream of interruptions. Our phones beep day and night. It’s either PagerDuty forcing our attention towards a particular infrastructure component or Zendesk telling us about a customer in urgent need of some TLC. In ops, there’s just no chance of getting something done without continually being interrupted. At least that’s what we thought.

Burning out

End of last year, things got truly bad. Other managed hosting providers were getting ahead on features, and we responded by launching catching-up projects. Meanwhile, technical debt kept growing, so we also started a cleanup initiatives.

Unfortunately, we didn’t get far with any of these goals. Outages forced us regularly to drop everything and make sure we keep our uptime promises. In turn, all this firefighting brought to the surface the fact that our people did not take enough time off. Because of our workload, some team members had kept postponing their vacations to the breaking point. With our capacity reduced by emergency recovery time, not only did overdue changes get even more delayed, but the strain on the remaining team members also increased. Adding insult to injury, customer feedback made it pretty clear they too noticed that we were great at starting projects but terrible at finishing them.

You can probably guess the effects this had on team morale. People were giving their best and had nothing to show for it. Everyone faced the same dilemma: Double down and risk burn-out? And what if even that doesn’t work? Or better draw a line, get some rest — and let the team down, maybe even cause irreparable damage to the business?

The coin drops

In January, I read an article in which Marten Mickos (of MySQL fame) says:

“The success of a CEO is measured by the results delivered.”

This statement made me realize that all of my many leadership efforts are moot if they don’t lead us to deliver what customers expect from us. And I finally admitted to myself that what caused our problems was not that people on the team didn’t know important from urgent, that they succumbed to scope creep, got lost in the details or did not take enough time off. Our failure was my failure.

A look in the mirror

My next step was to grab my favorite DevOps book — “The Phoenix Project”. When I picked it up for the first time, right when it came out in 2013, I found the similarities between its story and my former corporate home pretty hilarious. This time though, I recognized that the novel described… us.

• Team members were actively taking on too much work.
• The team member most familiar with a topic instantly became the bottleneck.
• Unplanned work kept disrupting the flow.
• We did not have nearly enough visibility of work in progress.
• We had no clear way of deciding what tasks we’d tackle next.

And the book not only described our problems, it also pointed out how to solve them! First, we needed to maintain our focus by visualizing and, even more critical, limiting our work-in-progress. For longer-term planning, we had to find an efficient way to prioritize both business projects and internal initiatives. And to get our work finished quickly, we had to prevent unplanned stuff from getting in the way all the time.

We found solutions for all these issues. In the following parts of this series, I’m going to describe them in detail.