Designing for Total Failure - Part 1

John Fahl
John Fahl
Jul 11, 2018 · 6 min read
I’m sure success is hidden on one of these tickets!

I wish I could tell you this was a happy story about IT conquest. You know, the pursuit of doing cool things with big stakes. It’s not. It’s a story of warning, abysmal decisions, megalomanical personalities, and how to burn $150M like they were scratch-offs.

Just a few disclaimers up front. This is a completely true story. This will be a multi part story. How many parts, I’m not sure. I’ve changed names to mostly protect myself but I’m sure a few of their egos need the cushion of anonymity.


It all started on a beautiful autumn day a few years ago. I’m sure birds sang majestic melodies and children played their silly games. But where I worked, we were about to embark on a super sh*tstorm and I didn’t even know it.

I was pulled into a conference room for a kickoff meeting to work on a “modernization” project. I didn’t know much about it, except there was a budget of $75M over five years — so it sounded pretty serious.

There were three of us engineers in the room. I was the Windows engineer. The other two I didn’t really know, a Linux engineer named Jeff and an Application engineer, Han. Han was quiet. I’d seen him around the office before, very smart guy but didn’t say much.

In came the architect, Shinesh and the Project Manager (PM), Pedro. I’d never met these guys except I had heard about Shinesh, or Shi. Some of the folks at the data center (DC) referred to him as a genius and a guy who was always tripled book for meetings.

Pedro tells us, “You guys should be touching yourself with excitement. (Those were his exact words.) We have executive sponsorship to fund this big project to go to the cloud, and you three are going to be the ones to build it in AWS.”

AWS preparing to mock us for our arrogance

This did sound exciting. Who wouldn’t want to be part of this project! It was at this exact moment when a giant f***ing cockroach climbed up on the table and proceeded past our notes and the phone like he was invited. I should have seen this portent and just put in my notice, but, like an idiot, I didn’t.

I said, “Sounds great! What are the requirements?”

“There aren’t any,” Pedro said.

Let me stop right now to tell you that I almost immediately filled my drawers. Let’s just say, that after working for major defense contractors and the federal government for many years, like Liam Neeson, I have acquired over my career a very unique set of skills to know that:

This. Shall. Fail.

Of course, I asked how could we possibly build something of this size correctly without any requirements. I told them that we would certainly fail. I said it day one and Shinesh shut me down right there. He said something along the line of, “Look guys, we are going to get this done together. There are requirements but many of the details are being worked out. For now, I need you to build the services to support all the applications.”

Without getting into the details too heavy, he was referring to things like Active Directory, patch and AV management, etc. Naturally, those items led back to the need for some information to be able to properly scope the build out.

Do we extend DC services?
Do we build new?
Do we build different?

We got no direction other than build me whitepapers with pros and cons of each uses case and “I’ll make the decisions.”

What I didn’t realize at the time, was that this was Shinesh’s MO. Architects are the only ones who truly understand the big picture and what needs to get done. In his mind, engineers are simply monkeys with keyboards to execute those tasks and incapable of making strategic decisions.

I brought up that at least “I” didn’t know AWS and looked to the other two, seeing they were in the same boat as me.

Shi continued with his pearls of wisdom, “If you don’t know AWS, I suggest you get on YouTube and start watching videos. That is all I did, I have terabytes queued up and have learned plenty.”

In my complete rudimentary understanding of how successful projects worked, it was then that I mentally confirmed that a concoction of youtube videos and zero requirements was a potion of slow — but certain — death.

I went back to my desk with the two engineers, and man, we were worried. We were all senior enough to know that we were in for misery. I think it was a genuine concern driven by worrying that whatever we built we would have to scrap it, or rebuild it, or rebuild it 10x because management didn’t know what they really wanted.

Pedro came up to my desk after our initial belly aching. I reiterated to Pedro that none of us knew AWS and asked if we could get training. He told us they were looking into training but it probably wasn’t going to happen. #HugeMistake

Of course not, I mean, why would they train us in something we all don’t know?

Pedro told me about a daily roundup meeting at 8am with the CIO and all project stakeholders. He said I should call in tomorrow in case they wanted to talk to someone from our team. I shrugged and agreed to it. My two fellow engineers shook my hand, I could tell I liked them already and knew the trenches were in front of us.

The next morning I dialed into the call. It was an intense 20 minutes as I heard the CIO asking for status reports from folks. They were shotgun style, a minute, two tops. The discussion was all strategy talk when suddenly the line breaks and Pedro mentions we had a kickoff meeting and that I was on the call to give a status update.

The line was silent.

I finally said we were going to research a few details before presenting a white paper and designs for the infrastructure build out. I omitted the fact that it would be all guesses since there were no requirements.

“When will it be done?” the CIO asked.

“We will be finishing it as soon as possible,” I answered. It seemed reasonably on par with their concrete planning.

“Oh my god, someone talk to this guy …” he started, “look this is a status call. Everyone comes on here and commits to dates and delivers on those dates. This is the cadence. This is how its done. So, when will this be delivered?” he demanded.

“Thursday,” I said. It was Tuesday and I wasn’t sure I could complete it in time but since i had zero prep time and one second to answer, that is what I said.

“Good.”

The call wrapped up a minute later and I realized I was put on the spot by our PM, just to be shot by the CIO without even so much as a thanks.

It was a glorious start.

LEARNINGS

Start with a plan! Does this need to be said?

Plans are worthless. Planning is everything. ~Pres. Dwight Eisenhower

Don’t shoot the messenger. Whether you’re a CIO or a junior admin, talk to people with respect if you want them to be open and honest with you.

If you enjoyed, this, try Part 2! https://medium.com/@jvftuo/designing-for-total-failure-part-2-907cd988ca6

I can smell an epic sh*tpile stewing right here, buddy. Unlike you, I will survive it.

John Fahl

Written by

John Fahl

I write stories and solve problems. DevOps guy, writer, startup co-founder, probably a sorcerer.