Somewhere within 20 feet, It’s a safe bet to assume there’s a cell phone nearby. That device, with all of its memory, CPU registers, and pixels, is roughly 100x as powerful as what men and women used to send humanity to rocketing toward a small rock a few hundred thousand miles in the upward direction.
I look back on NASA, and I recall all of the achievements that made that final step for mankind possible. The Gemini, Mercury, and Apollo missions that came before them each solved a major problem, by experimentation, that space flight was possible. It’s hard to imagine now that people had no idea what would happen when you put a human being in space. Would their head simply explode, and if so, what kind of velocity would an eyeball have at absolute zero? This was a question that was no doubt asked by a bunch of incredibly smart people trying to plan out how we’d be able to safely send someone into an incredibly tight window between being flung back to earth in a ball of fire or into an infinite abyss. Could you imagine an off-by-one error in the math, or an unhandled exception? When was the last time your programs had those problems?
If you’re a modern day software developer, you’re either reading this between fixing those kinds of issues, or writing tests so those errors stay fixed. For now.
The terrible truth is, no matter who you are or what sort of systems you engineer, be they web pages or embedded software, you can often tread the water between catastrophic failure or lucky circumstance.
If you’ve worked on any software project with more than 10 individual developers, I’m almost certain that there have been moments where you’ve contemplated just throwing it all out and starting over again.
Why is this so common, I wonder? It’s not like the individuals are bad programmers. In the past 12 years I’ve met some absolutely incredible programmers who’ve been able to teach me things that I’d never have thought of. They’ve expanded my view on systems design and how they are designed and constructed in both paper and personal contexts. I respect them greatly. But the fact remains, that more often than not, a system tends toward a giant ball of mud given enough time.
Brian Foote and Joseph Yoder, in the paper “Big Ball of Mud” assert that it’s when a combination of pressure, risk aversion, inexperience, complexity, rate of change, and scale of a project all increase, so too does the big ball of mud.
The evidence for this — to me anyway — is that there are a lot of inexperienced developers out there in very senior positions in our trade.
Some of that is no doubt due to the culture of the web and its construction crew being such a new industry. Unfortunately, I don’t know how well it’s maturing sometimes. It can still feel like it’s people in suspenders and thick rimmed glasses banging unmentionable things into terminals and posting on usenet. We’ve just traded out the terminal for a MacBook and usenet for Twitter and Facebook.
I’d like to stress right now that I don’t have all the answers. I’m not the worlds best programmer and frankly, there’s an infinity of things I could be more knowledgable of. I am average in a galaxy of incalculable talent.
I want to make things better, though, and I think it starts by emulating the behavior of an organization that I admire.
This image brought to you by preventable catastrophe, budget overruns, political wrangling, infighting, incalculable complexity, and against its best effort, humanity.
Several billion dollars, decades of planning, engineering, and prototyping. Detailed, painstaking micrometer measurements and manufacturing precision, and at the end of it all the Hubble team shipped a broken system.
When the Hubble first turned on, one of the solar arrays that provided half the power for the thing didn’t unroll, which forced the whole system completely inoperable. Something around the size of a city bus would be junk, and its in space.
The astronauts on the shuttle who launched Hubble ended up preparing for an EVA — a spacewalk — to manually unroll the solar array. After a harrowing few minutes, brushing with complete system failure in production, they found a way to patch the system, forcing it to unroll properly. I’ve never read an exact accounting of the error that caused this, but from what I’ve watched and read, it seemed like a bunch of programmers huddled in a conference room mapped out the system looking for commands that they could use to ‘override’ the original method.
But in all honesty, this was mostly a warm-up act for what came when they started getting the first images back from Hubble. Nothing was in focus. Every picture they saw had a corona effect on any light source, which for a telescope, is basically certain death.
The underlying defect was a problem in the main optic of the telescope, the giant mirror that gathered light. It contained a defect of effectively point zero zero zero zero zero one inches. that’s 0.000001. The fate of a system that costs billions of dollars rested on a number that seems almost imaginary.
So why did this all happen? Was this just bad luck, or a combination of factors with an organization bereft of standards and rigor? Were contractors involved?
The primary mirror is a disc of glass 2.4 m in diameter, whose polished front surface is coated with a very thin layer of aluminum. When glass is polished, small amounts of material are worn away, so by selectively polishing different parts of a mirror, the shape is altered. During the manufacture of all telescope mirrors there are many repetitive cycles in which the surface is tested by reflecting light from it; the surface is then selectively polished to correct any errors in its shape. The error in the HST’s mirror occurred because the optical test used in this process was not set up correctly; thus the surface was polished into the wrong shape —The Hubble Telescope — Optical Systems Failure Report
The design of the telescope and the measuring instruments was performed well by skilled optical scientists. However, the fabrication was the responsibility of the Optical Operations Division at the Perkin-Elmer Corporation (P-E), which was insulated from review or technical supervision. The P-E design scientists, management, and Technical Advisory Group, as well as NASA management and 1 NASA review activities, all failed to follow the fabrication process with reasonable diligence and, according to testimony, were unaware that discrepant data existed, although the data were of concern to some members of P-E’s Optical Operations Division. Reliance on a single test method was a process which was clearly vulnerable to simple error. Such errors had been seen in other telescope programs, yet no independent tests were planned, although some simple tests to protect against major error were considered and rejected. During the critical time period, there was great concern about cost and schedule, which further inhibited consideration of independent tests.
The final report slammed both NASA and the optics contractor, Perkin-Elmer. The commission found that while manufacturing, quality control issues misaligned the instrumentation that measured the mirrors for correctness and nobody else assumed there could possibly be an issue. So off they went to space. NASA had a heated relationship with Perkin-Elmer, so much so that the report called out the contractor for regular schedule slippage, cost overruns, and quality control problems.
I just call it by what it was: A big ball of mud.
I have no doubt that you could imagine a similar situation. You don’t have wrinkles on your forehead and gray freckling your hair until you’ve witnessed a software project melt down at the precise time it shouldn’t. It’s not so much of if it’ll happen, it’s more of a when.
Software running on the internet, especially in the consumer application and user interface world tend to follow a similar pattern. One level down, and you start to question how this has worked so well for so long. Is it me? Do I lack the essential programmer-stuff that lets me see the vision in how this all orchestrates into the output of a true free electron’s thought patterns that never grew out of the prototype phases?
I’d love to be surprised. But in the real world, I’ll sulk into my chair and mumble under my breath about not having enough time or energy to pour into fixing the problem. I have kids to provide for. I have some semblance of a life that I want to live outside of the boundaries of my office.
The other terrible truth that nobody tells you is that if you want to really be the best at what you do, in your profession, it takes endless patience and consensus building to guide your peers into doing it the Right Way. No single person can be the bedrock of a sufficiently complex system. It just isn’t realistic to expect a few people to steward a project into something that I’d want to strap into and let it carefully explode so that I could travel hundreds of thousands of miles to walk around a desolate wasteland that could kill me at any second.
I find myself mentally exhausted and emotionally drained after a long day at work. It’s definitely not a new feeling, and it’s something I’ve struggled with for a long time, professionally.
People can be said to be perfectionists, but in the software world, perfection is simply being correct. There’s no shades of gray about being correct or incorrect. The computer doesn’t care much for nuance. You are either correct or you aren’t.
The mental exhaustion comes from working out every possible input and output to a given activity, and then translating that into a very specific an precise instruction set for something that amounts to the intelligence of a wet blanket. Unless you are providing correct instructions at the exact points in time, things end up going from usable to a nightmare. Insidiously, this often happens in subtle ways that are easy to miss.
The overwhelming emotional response to this metamorphosis of bullshit is where I find a lot of programmers at, day to day. “What idiot would do such a thing?” is a common complaint.
They’ll blame the APIs, they’ll blame obtuse and brittle interfaces. They’ll blame absolutely everybody — including the testers— for having work in an environment so vast and broken and functionally idiotic that they just can’t write Good Code.
Frothing spittle and twitching at the eyeballs, they’ll be lurched over their keyboards, four in the morning, looking through the same file over and over again, manically searching for any potential problems before giving up moving on to another and repeating the whole vicious cycle.
This is insanity. The way we build software right now is completely fucked.
I’ve been giving this a lot of thought, and I’ve been doing scores of reading. I don’t have an answer, but I think there’s a sort of system that I tend to follow, and when I see somebody deliberately break out of that system, It drives me up the wall. I want to stop them and ask them why they’re doing things so badly. Does this make me a dick? Shouldn’t we stop when we see our peers doing something ridiculous or dangerous or just flat out incorrect and guide them toward the Right Way?
At nearly every job I’ve held in the last decade, I’ve seen people breaking out of this system, including myself. Sometimes it’s out of expediency, sometimes it’s because this is just a prototype, and that’s okay. Prototyping is what Good Coders do, right?
More than I’d like to admit, its been nothing more than simple procrastination.
Procrastination is a symptom that all software developers have. It’s acquired over each and every accomplishment they’ve had along the way to this point in time. Every time they’ve shipped some Good Code, there’s a drip of hubris into the bucket of egoism that they’ve just become this walking behemoth, capable of moving mountains.
We just can’t help it. It’s the Makers High. We wouldn’t put up with countless endless nights, souped up on lethal levels of caffeination and sugar, working to the sounds of completely ridiculous music that we believe get us into flow and make us the most productive.
Really though, all this ends up doing is detracting from what we’re good at.
Developers, Engineers, Coders, Hackers, whatever you want to call them — They’re all the same — they’re all analytical and egotistical. The systems we forge are born out of a thought exercise and grow organically out of the requirements as we know them to be — in the moment — and as a group, we don’t tend to backtrack unless it means coming to terms with the fact that we were wrong.
It’s not like we just forgot to get the milk from the supermarket or left our wallet at home every once and a while. It’s like creating Frankenstein and then, after you’ve hit it with lightning and given it life, while attempting to walk, you realize that you didn’t bolt the legs on.
That’s one reason engineers can be finicky, difficult people. We’re shoved bug reports and defects in our faces like they’re just some paperwork we have to do. There’s significant psychological subtlety in place that most program managers don’t really understand, since most haven’t had that feeling of building the thing and forgetting the detail that breaks the thing.
Unless you’ve written a buggy program, you don’t realize that you’re addressing our intellect. This is why I think that every engineer on the planet looks at a bug report and feels a twinge of pain as they read whatever detail that was left to serve as a figurative shame sticker on the report card of their creation. It really sucks when you’re just flat out wrong.
Being wrong — rather, being incorrect — is an extremely humbling experience. The catastrophically incorrect, which is when software crashes, money is lost, or the absolute worst, data is stolen, is the kind of thing that makes you question your career choice. It makes you want to curl up into a ball and weep at how completely fucking stupid you were when you’ve found the problem.
But what you should have had was a swift kick in the ass and a system that prevents you from being the egotistic shit-for-brains you were.
What follows is my personal system. I try to follow this as best as I possibly can, and use it as a means for checking my ego at my editor. There’s no silver bullet, so before you disagree with anything here, take a moment to ask yourself if you’re technically correct, or just trying to sound smarter than I am. These are rules that I personally follow, and not everyone is going to agree with them. You’re wrong, though.
Well, actually — you’re incorrect.
Code doesn’t change. It mutates.
A healthy codebase is one that is thriving with life and activity. Much like the genetics of any living thing, it requires constant mutation to evolve and grow new capabilities. The cost of all of this mutation can be messy, and if not carefully checked, inaction can lead to death.
Dead code — unused, old, brittle, or broken — must be expunged early and often for the continued health of a system. Commenting out and committing doesn’t count. Refactoring and not updating documentation or tests is a cardinal sin.
If you’ve ever played with a big box of lego bricks, you’d probably find that unless you were following specific directions, there’s a decent chance that your awesome spaceship ended up with a few weird appendages or blocks that were out of alignment. Over time, unless corrected, those blocks end up as a permanent blemish on the overall structure.
Our codebases are no different. Perfection isn’t the goal, it’s the baseline.
What defines, perfect, though? Perfect is the enemy of good, isn’t it?
I think simplicity is perfect. Simple is elegant. Simple works.
Complex solutions to problems show defects in our comprehension of a feature, or of how that feature works within the larger system. When you feel like you have to retrace your steps when working with a file, that’s a sign that your code is getting too complex, and you should think about refactoring, time permitting.
The more complex your solution gets, the more mutations that it will have over time, and the more likely it’ll become the root cause of a problem rather than the fix to another. Simple isn’t sexy.
You need to have a functional vocabulary.
You write software for two specific groups. The compiler, and the consumer. The consumer for what you write is other developers. Thus, agreeing to idioms in communication are key in order to participate in the discussion.
Don’t really worry about the compiler. It’s basically a blithering idiot. Assuming that a compiler were clever would remove the need for our job. Developers, though— that’s another story.
Imagine for a moment that you’re sitting in a meeting at work. People are talking about a new feature that needs to be implemented, and you are tapped to make the feature and push it live, all within a week.
People are throwing around acronyms and words that you don’t know. You’re handed a diagram of what you can only assume are a tube sock and a bottle of nail polish, and that’s it, you’ve got a week. Good luck.
So now you get to run head first into a project that you’ve never worked in before and figure out how to tell it what tube sock is an how to make it this shade of nail polish in a week.
Senior developers that I’ve worked with who were worthy of the title are able to find similarities into objects and move those from ‘gym sock’ into ‘tube sock’ and go from that point. Less senior developers tend to try to tell a dumb compiler exactly what a ‘tubè söck’ looks like. Even less senior developers attempt to find a tube sock online and copy and paste that into our project, only to spend the other 4.99 days figuring out why the tube sock looks more like a thimble and is blue.
People have argued with me, at length, that their code is readable. Readable code therefore is elegant enough for any common developer to read and understand how it is used, and in which context it is used.
Unfortunately, reality is different. Your ‘readable’ code is just hubris. You may write a feature that may not be documented, and may not have gone through a refactoring phase and got cleaned up to standard because it’s ‘readable’.
So really, your readable code flies in the face of standards. All it does is throw the reality of the situation into the other programmers who have to maintain that feature in the future — perhaps when you’ve long since left — and they have to decode not only the instructions for the computer but the instructions for humans.
Be simple. Stay small. Ruthlessly think about what you’re creating or modifying, and make a note of it in docstrings or full documentation blocks. Don’t be the idiot that says var foo = bar; // bar is foo because you’ll be ridiculed. Tell yourself what you’re going to do, and then implement that sentence. When you’re finished, take that sentence and write a test, then just move on.
Review code as though you’ve never seen the system it exists in. Does it make sense? Overly esoteric classes only work for the extreme cases, and only with the proper ridiculous and long comment explaining the straying from the norm.
// Dear maintainer:
// Once you are done trying to 'optimize' this routine,
// and have realized what a terrible mistake that was,
// please increment the following counter as a warning
// to the next guy:
// total_hours_wasted_here = 42
//When I wrote this, only God and I understood what I was doing
//Now, God only knows
/// Class used to work around Richard being a fucking idiot
/// The point of this is to work around his poor design so that paging will
/// work on a mobile control. The main problem is the BindCompany() method,
/// which he hoped would be able to do everything. I hope he dies.
* You may think you know what the following code does.
* But you dont. Trust me.
* Fiddle with it, and youll spend many a sleepless
* night cursing the moment you thought youd be clever
* enough to "optimize" the code below.
* Now close this file and go play with something else.
At this point, I'd like to take a moment to speak to you about the Adobe PSD format. PSD is not a good format. PSD is not even a bad format. Calling it such would be an insult to other bad formats, such as PCX or JPEG. No, PSD is an abysmal format. Having worked on this code for several weeks now, my hate for PSD has grown to a raging fire that burns with the fierce passion of a million suns.
If there are two different ways of doing something, PSD will do both, in different places. It will then make up three more ways no sane human would think of, and do those too. PSD makes inconsistency an art form. Why, for instance, did it suddenly decide that *these* particular chunks should be aligned to four bytes, and that this alignement should *not* be included in the size? Other chunks in other places are either unaligned, or aligned with the alignment included in the size. Here, though, it is not included. Either one of these three behaviours would be fine. A sane format would pick one. PSD, of course, uses all three, and more. Trying to get data out of a PSD file is like trying to find something in the attic of your eccentric old uncle who died in a freak freshwater shark attack on his 58th birthday. That last detail may not be important for the purposes of the simile, but at this point I am spending a lot of time imagining amusing fates for the people responsible for this Rube Goldberg of a file format.
Earlier, I tried to get a hold of the latest specs for the PSD file format. To do this, I had to apply to them for permission to apply to them to have them consider sending me this sacred tome. This would have involved faxing them a copy of some document or other, probably signed in blood. I can only imagine that they make this process so difficult because they are intensely ashamed of having created this abomination. I was naturally not gullible enough to go through with this procedure, but if I had done so, I would have printed out every single page of the spec, and set them all on fire. Were it within my power, I would gather every single copy of those specs, and launch them on a spaceship directly into the sun. PSD is not my favourite file format.
Assume that every developer who has to open this file has a picture of you and your family, your address, and a vendetta.
Documentation is the encyclopedia of your mutations.
I absolutely do not believe a single person who says that their code is self explanatory. I firmly believe and will happily defend this position. Anything less than function/method/class/module level comments and you’re just lazy.
I’m still shocked that this is considered a hyperbolic position to take. People give you money for your creative writing output, and yet, you refuse to do any of it outside of the absolute baseline. Basic. For the sake of those who are your brothers and sisters on the front lines, make their lives easier.
A few lines of comments describing the contents of a file are far more likely to impress anybody than any clever optimization you may attempt to do beyond them. How often have you looked at a library that was well commented and remarked, “Oh hey, this even has comments!”.
Documentation is critical to prevent the signs of mud growing on the walls of your project. The encyclopedia of mutation gives everyone access, and equality of knowledge on how to use and mutate it further. It’s the importance of sequencing the human genome. Why would you not want this?
Having a dictionary that’s well organized is as essential as the dictionary itself. What language do you write in? It doesn’t matter — it’s got an automatic tool that turns docstrings into a webpage.
Automation is essential.
Every single time you stop writing is time that you’ve wasted doing something that you shouldn’t have to do.
Make is one of the oldest programs on the planet, yet only recently in the UI world, we’ve begun to automate the most tedious of tasks.
Automation is how you can grade everyone’s knowledge of the vocabulary and the rules of the grammar of the system. Linters, tests, continuous integration and automatic documentation generation are things that take effectively fractional margins of time compared to the lifecycle of the project and give them back to you.
Why is it that automation isn’t something everyone would want? We’re lazy, as I mentioned earlier, and if we can avoid it, we shouldn’t need to do it.
Regardless of what you’re building, tools exist that can automate vast portions of overhead from your project. Spend the time to invest in the scaffolding of what you’re building, and you’ll be safer in the long run.
Prototyping! —I get it. Who wants to do anything automated as a prototype? Well- the best of the best tend to automate their automations. Boilerplates, empty repositories— Blanks — are everywhere, especially in the web. Artists would get nothing done if they had to hand stretch every canvas and make their own oil and acrylics every time they needed to paint, so why do we do this to ourselves as developers?
My basic rule is this: If something is performed more than three times in a 4 hour span, automate it. No exceptions. Assuming this task is 10 minutes of effort each hour, that’s an aggregate of 3 hours saved every week. 14 hours a month, and so forth.
Gravity is my secret weapon.
This is something that speaks to my lifetime battle with debt. Dave Ramsay, a guy who’s ideas I enjoy, has a theory about paying off debt as fast as possible.
The theory is, paraphrased, that if you focus on the smallest debt with all of your energy first, and get it completely paid off as fast as you can, while paying the minimums on every other debt you have, you can focus all of the effort from that onto the next one.
As you continue up the chain to the largest debt, typically mortgages, cars, and so forth, you’ll be better able to make a dent in the principal of the debt with larger than expected payments.
For the astute reader that has ever dealt with crushing amounts of technical debt, this theory might give you pause for a moment.
A good credit rating for a US citizen means that they have more affordable means of borrowing, and they can get more out of their money than folks with a bad credit rating, if at all.
Why should your codebase be any different? If new features and refactoring effort were credit, you’d want to be able to maximize how much your effort will get you before having to pay off the debt.
This is the snowball effect. Use gravity to help your snowball grow from a few minor annoyances to shipping an entire new suite of functionality quickly and deliberately.
Your peers are your students. Your code is your syllabus.
Take every opportunity you can to educate your peers about what your code does. Are there certain ways to interact with your interface? Do I need to have an instance of this or is it new-able? is there a pattern to how this file is constructed? Is there a private or public interface?
Code reviews are such an excellent opportunity to flip the switch for your co-workers.
All too often, I’ve seen code reviews go from the ‘The English Patient’ to ‘The Spanish Inquisition’, and frankly the environment is one of the most important — if not dangerous — interactions you have with your team.
Critically, code reviews give you an opportunity to show competence to your peers. It’s a five minute show and tell about what you’ve been doing. It shows that you’ve been catching on to the language of the project and can be fluent in it. It helps you build trust and show off your understanding of the relationships of the vocabulary.
I know you have no time. There’s no budget, there’s no flexibility in the features for launch and you certainly wouldn’t want to spend a day sitting in a conference room when there’s code to be written.
Do it anyway. Do it with ferocity and attention. Make time. Be measured, back up your arguments with fact, and whatever you do, do not utter the words “Well, actually…”. If you do, you’re out of the meeting.
Egotism and Defects are interdependent.
My final rule is one burned into my mind from a path of scorched earth I’ve left behind me.
Experience is the tincture we apply to failure caused by egotism. The most incredible developers I have met in my career have been the some of the most decent, humble, approachable, jovial individuals I’ve had the pleasure of working with, without exception.
You’ve probably met some of these people, or at least I’d hope you’d have the good sense to seek them out and emulate them in their behavior. These are the people who aren’t politically afraid to ask if you actually understand something, and then spend the afternoon over beers and wings explaining just how they interact with one another, and why the decisions were made to build them this way.
They are the absolute backbone of any engineering culture. They are the vault of institutional knowledge of any company. Befriend them. Learn everything you can from them.
Egotism is selfishness in the guise of cleverness. Brevity is the soul of wit, in english, perhaps, but in code it can be a dangerous cloak to complexity or a lack of comprehension. Brash, dismissive developers are almost to the person hiding a lack of self confidence or are so absorbed in Egotism that they simply cannot believe they are wrong. They are. Go look at their defect counts over the last year. Bring an umbrella when you open the door, it’ll be messy.
Try real hard not to be an asshole. Nobody wants to work with someone who thinks everything they’ve written has been brilliant. We’re generally a self deprecating people. When you tell us that your shit doesn’t stink, we automatically assume it does and will start smelling it by default.
Being humble doesn’t mean that you can’t revel in your successes. It just means that once it breaks — and it will — you just get right to work on what went wrong, chuckling at the defect title being an obtuse little sentence, and discuss lunch options.
Never assume your code is finished. Remember; code mutates. Mutation happens with or without direct influence. Document it, and make the mutations as simple as you can, but no simpler. Don’t be a dick. Try to teach others about your decisions for each mutation.
Your peers require your respect, your attention, and your effort. You are the best person to set the example.
Now — What’s for lunch?