The worst project I ever worked on
Taking NIH to a whole new level
I was recently asked, “what’s the worst project you ever worked on?”.
After a quick scan through my mental graveyard of half-finished, half-assed ideas too embarrassing for me to dignify with a link here, I settled upon my first job out of university.
I had been hired as a Software Engineer by IBM, to work out of their main UK development laboratory, Hursley Park, home of the mighty CICS, WebSphere MQ and MessageBroker products. These are the gold standard in messaging and transaction processing backends that do the heavy-lifting behind the scenes of lots of Big Enterprise Systems.
However, my role at Hursley was on the WebSphere MQ build team: the team responsible for taking the big MQ codebase and compiling it on all the different supported platforms. Linux on several architectures, Windows, AIX, HP-UX, iSeries, zSeries, …
It was an operational challenge managing jobs in wildly heterogeneous environments and getting the right information out when compilation failed. The team had originally used a bunch of cron-style jobs on each build machine, which, a couple of times a day, would pull down a cut of the code, compile it and make the results available, on a network share I think.
By the time I arrived, however, that had been replaced by JEM - the Job Execution Manager. JEM was a central server, communicating with agents on each build machine. It meant that builds could be started on arbitrary machines, at arbitrary times, with arbitrary changesets; it also gated the jobs so that build machines didn’t get swamped with several at once.
JEM was kind of A Big Deal and, scuttlebutt had it, had launched the guy behind it onto a promotion hyper-fast track.
However, despite JEM only being two or three years old at that point, it had already become something of a black box to everyone on the team, as people had moved on.
If something went wrong, “turn if off and on again” was always the first (and sometimes only) strategy: there was no point trying to capture information about the broken state, because no one could take action on it.
Adding new machines into the build pool was a black art, and nobody ever seriously considered the server-side changes required to support the continuous integration environment that the developers were asking for.
So why was this the worst project I ever worked on? Every developer is familiar with Not Invented Here, and has probably been responsible for it at some point - I know I have.
The reason JEM is so egregious to me, is that it actually already HAD been invented here. Let’s look at some of the requirements:
- jobs to be orchestrated from a central server
- reliable communication between server and clients
- extremely heterogeneous client environments
- build requests to be queued until client is ready to accept them
Does that remind you of anything?
How about WebSphere MQ?
A team of four people was required to administer and manage JEM, when the very thing it was building had been the perfect solution all along. It should have been self-hosting!
Rolling your own janky solution when an established product already exists to fix the problem is a poor show, but understandable and super common.
Still rolling your own solution when the established product is well known to you, you can use it for free, and with effectively unlimited support from the world’s top experts, is inexcusable.
Still rolling your own solution when the tangible output of your daily work is itself the established product (which you can use for free, with unlimited support)… that’s the worst project I ever worked on.
