Digital disasters — a cautionary tail

Nowadays, It seems quite rare to read any articles talking about digital disaster projects. Most of the pieces I come across seem to be dripping with happy blue sky smugness. However, let’s face it, in the real world 9 out of 10 digital projects that occur will have a few issues, ranging from mild spring showers to tropical storms with rather dull names and all the way up to natural disasters that threaten the very nature of reality. And number 10? The tenth project will be the one that actually kills off the dinosaurs.

Today’s bedtime story concerns a project situation I assumed control of several years ago and which had all the hallmarks of success. As I took over, all the project deliverables had already been signed off, there was documentation leaking out of the woodwork and we were in production with an experienced team and with plenty of time and resources to hit deadline. Well, that was my initial opinion based on the first few days.

And then I started to really read the documents and they were an absolute dream, albeit a very nasty one involving people called Freddy and knives and bloodshed; it meandered crazily between far too much information on irrelevancies all the way up to abstract art on areas where detail would actually have been rather handy.

Documents such as DFD’s, journeys and stories, ELH, technical infrastructure, integration documents, API definitions … Let’s just say the list of missing documents was extensive.

UX was still in its infancy back then, a.k.a. non-existent, methodologies were still clearly dropping off a cliff with no idea as to what reception they would receive at the bottom, even though time and budget had been signed off.

My first foray was to question a few very basic but crucial usability errors in the project requirements document — i.e. the main piece of work that was being used by our UI resources to build the front end. At this point I was informed by the client partner that whilst we might be able to squeeze some of the critical changes proposed into the timeline, it would have to be on the down low and I was to treat the existing documentation as written in stone.

Next, I discovered that this set in stone project was in fact still being driven by design, with no technical sense checks, no budget changes to the documentation and a visual designer who was drastically changing the interfaces without any safety net. The developers, I was informed, would just have to work off new annotated designs. So, instead of a designer working off approved and signed off UI/UX wire frames and working with an approved style guide, she was instead redefining the whole project with unlimited client iterations. Madness.

I didn’t think it could get worse until I started looking much more deeply under the hood at the back end operation. This is when I discovered that all of those wonderful technical documents mentioned earlier were instead living a very bizarre virtual life inside the head of the primary technical lead. This guru, and he was brilliant, was unfortunately not that great in the planning and estimating department.

The one positive aspect, or so I thought briefly, was that the client was very, very understanding about the continuous design amendments having a financial impact on UI development — seemingly throwing wads of cash at us for simple design changes. The only problem I discovered was that this cost, didn’t even come close to the true total cost of ownership inside the project.

In this case as design was driving us toward the cliff, components were being created, modified or cloned and as a result the back end would have to be modified to allow for this, the HTML/CSS/JS would have to be changed and re-tested, the project definition document needed to be amended to track the changes which also meant an updated test plan as well. Finally, the overall project plan and Gantt would have to amalgamate all of those changes. In other words, those 2 days quoted by the technical team to represent the updated designs actually represented about 20% of the true cost — and these extra costs were just getting bigger and bigger.

I was in trouble and I must admit to having had a real urge to do a Titus Oates, pretend I was in the Antarctic, and just to go out for a really, really long lunch one day.

So, what happened? Well, firstly, I did have a really long lunch, although I spent most of it disaster relief planning. I knew the asteroid was coming, I just didn’t know yet how bad the impact was going to be and whether I could get any of us to safety first.

My first change was to stop any further design led changes from happening, and to do this I made sure to insert myself in between the designer and the client as a technical safety net. Actually less a net and more like a concrete wall, reinforced and with razor wire on top. And mines, lots of mines. If at this point any further changes were really required, I made sure the cost reflected reality.

At the same time the UI team was set to re-estimating based on that snapshot in time, using approved but changed designs and updating the requirements document, which in itself had to fed to the testing team for updates.

The most important step was to manage the planning of the back end. In this case we were in a situation with virtually no documentation but one very smart guy with lots of beer mat estimates. We sat down together and we took a week to go through a big list of what I thought should have been in the original documents and why. We talked, we argued, we laughed (briefly but ironically), we drew diagrams with lots of arrows, we made more notes and, it must be said, I drank a lot. Based on this work I then waited for our technical guru to come up with the answers, the diagrams and the numbers. A bit like an expectant father, all I could do was hover anxiously.

After a week we discovered the project probably would be 3 weeks over on the estimate. I could handle this, it was even within the scope of the overspend if we sucked it up and used an extra in-house resource. Another week went by, more diagrams appeared as the evaluation continued. The delay then turned into 6 weeks, and soon after 10.

At this point I had to bring in the client partner and the directors and suggest that whilst we had a major disaster of a project that was going to be 2.5 months late, it was more probable, in my humble opinion, that the delay would more likely turn out to be double that.

The only solution was to literally suck it up, be as direct and honest with the client as possible; humble just short of reaching for sharp Japanese implements of death, and try desperately to rebuild trust levels.

At the end of the day we delivered the solution about 14 weeks later than originally planned, this involved bringing in 2 extra external resources and a lot of late nights and stress. That wonderful client trust, unfortunately, was never recovered.

Now, a lot of you out there will be clinging smugly to your new agile UCD process and exulting that this could never happen to you. I would counter by suggesting that a process is only as good as the people inside of it. The truth is that even if your process is watertight, it is still reliant upon the efforts of individuals, their skills and the inherent risks within your entire team.

If I was being slightly negative my advise might be to always embrace your paranoia, cherish your skepticism and trust no-one. However, if you think about what went wrong in the above tale, you should be able to see that every possible misstep could have been overcome by user centred design, effective team communications and some good old fashioned honesty.