What I’ve Learned From My Mistakes

Almost everybody has heard the standard job interview question “What challenge did you solve in your last position, and how?” or its variant “What did you learn at XYZ…?” While these can be useful windows into a candidate’s experience, what a person learned when everything went well is only part of the picture.

Things don’t always go well, and we all make mistakes — all of us. Mistakes, and the lessons we learn from them, impact how we approach new challenges, and I’ve now made it a standard part of my own interview process to ask candidates what their biggest mistakes were. This has led to some amazing conversations and new lessons for me, and I encourage you to try it as well.

It can be hard to talk candidly about our failures. Particularly in the technology community, we embrace a culture of success, and when we share what we’ve learned we tend to highlight our achievements and downplay our errors. But there is no reason to be ashamed of mistakes: they’re an inevitable part of growth, and everybody makes them. Anybody claiming 10+ years of software engineering experience who cannot recall, vividly, something that went wrong is:

  1. Lying to him- or herself (and you),
  2. Wasn’t smart enough to recognize the mistake and learn from it, or
  3. Doesn’t actually have the experience he or she claims.

I believe mistakes are incredibly valuable experiences that a well-rounded candidate brings to the table. They teach us caution, and they teach us to ask questions and probe more into “what might go wrong.” Double-checking backups, thinking twice about deployments during critical workload periods, getting key stakeholders on board in a new project… These all seem like good ideas, right? Well, to somebody who has been burned by one of these issues in the past, they aren’t just platitudes — they become instinctive rules to be championed at every opportunity.

It can be hard to talk about mistakes, especially during a job interview. Lead the way by sharing one of your own.

It is easier to acknowledge these truths than it is to share the details of our own failures. Even if we intellectually “know” what I’ve written above, failures can be embarrassing to discuss and candidates may be reluctant to share their best horror stories. I’ve found I get better, more honest responses when I start by sharing one of my own, and I encourage you to try doing the same in your next hiring cycle.

Here are three that I can recall vividly, and what I’ve learned from each.

The Buffer Overflow

In the mid-90s, when I was fresh out of college and very inexperienced in my career, my employer had a client in the food distribution service industry. Taking inventory was a critical part of their operation, but also a disruptive one because it required shutting down the warehouse and laboriously counting products by hand. Inevitably, human error crept in and recounting was required. We saw this as a great opportunity to pitch a new bar-code-scanning device, a hot technology at the time.

Unfortunately, I misjudged the quantity of data that would be scanned, and learned the hard way what “buffer overflow” meant. To me, a few thousand entries was a LOT of items, but for this client it was just a fraction of their actual inventory. This device and its SDK were incredibly primitive by today’s standards, so no compile- or run-time issues were reported, and nobody knew there was a problem until the end of the cycle when the data set came back short. A 2-day work shutdown was completely wasted.

This experience was my first exposure to buffer overflows, and I’ve watched for errors like that ever since. But my real lesson was about studying clients more carefully and gaining a deeper understanding of their business requirements before beginning to build solution. Automated testing tools and concepts did not yet exist at this time, but as tedious as it would have been to simulate the client’s actual usage by hand, I could probably have done it in a few hours with a good plan.

Always test with real-world data. Woodworkers have a saying: if you don’t test your finish on a piece of scrap, you test your finish on your workpiece.

The Hidden Customer

When “Y2K” rolled around, many IT firms got work helping customers audit and upgrade systems that could not handle 4-year dates. One of our contracts was to upgrade a local client’s antiquated accounting system that was straight out of the 80’s: green-screen terminals and all.

Accounting systems are a lot like religions — they share a common purpose, but implementations and workflows vary widely, and switching to a new one can be arduous and stressful. To ease the pain for our clients, we selected a product that (for a hefty price!) included the source code. This was incredibly rare in the pre-Open Source days, and we felt we had a winner because we could easily customize the system to suit our client’s needs.

We demonstrated our recommendation to our client, and they were enthusiastic. The entire project went well until a few weeks before we planned to deploy — August, 1999. At this point the client introduced us to the head of the Sales department. Also, a co-owner of the firm and related to the CEO. A critical internal user who was not previously involved with the project, and who had a requirement that had not yet been discussed. We were faced with a massive scope change with barely 12 weeks to go before the (literal) drop-dead date.

I was young and not very assertive, and I wanted to please clients so much that I said “yes” to basically every request. I also had more energy than experience, and as the young often do, tended to work harder and longer to cover my mistakes rather than admit I was wrong. I believe all team leaders and managers should watch for this tendency in junior team members, because while it is often possible to do, it is neither healthy nor productive.

I said yes, we could do it. Then I worked back-to-back 36-hour shifts for three weeks straight, and I pulled it off—just barely. There were plenty of bugs, and lots of spill-over to deal with in the following months, but I built this feature the customer wanted. Imagine my surprise when I wasn’t viewed as a hero after all this. The customer was unhappy, but instead of being unhappy with a faceless software vendor as usually happens, they were unhappy with me.

A large standard deviation in a service metric makes customers unhappy. An average delivery time for a refrigerator of 7 days is no good if the range is 2–14 days.

I learned several lessons here. First, the act of saying “yes” does not make one a hero. In this case, doing so created significant risk for the entire project, and the quality of the delivered product was lower because the work was rushed. The microwave had the magic button they wanted, but it was delivered at the last minute and the door didn’t always close.

Second, always have backup. Even CEOs have somebody they can fall back on if something goes sour — the board, the investors, the lawyers, somebody. Never feel as if you alone carry the torch for the entire company. As sports fans know, winning teams aren’t filled with solo mavericks — it’s solid teamwork that wins games. I should have asked for help the moment this wildcard was introduced.

Finally, always understand who your “real” customer is. Dig deeper until you understand the entire process. How does the customer do business and make money? Who writes the checks? Who will use that shipping module that everybody else said was a minor detail? Make sure everybody is accounted for and their voices are heard.

Trust, But Verify

I once took a contract-to-hire position at an entertainment company. It was a pretty exciting opportunity, but as anybody who has ever done this knows, contractors are often second-class citizens. You can be the “expert” on something, but there are always meetings you aren’t invited to, systems you can’t see, and “that’s not how we do things here” trust barriers.

Happily, I was working with a very competent internal team, a department head who was open to new ideas, and supporters from other divisions who really wanted the project to succeed. Unhappily, this organization had a very strong separation between Dev and Ops. (DevOps wasn’t a “thing” yet.) Developers produced software. Ops ran it. The story almost writes itself.

In launch week, a small but crucial issue turned up only in Production for an image-resize service that the site was heavily dependent upon. Dev had no ability to inspect the Production systems, so what ensued was hours of unproductive debug-by-email and finger-pointing. Eventually, a crude hack was introduced to work around the problem. It worked, but we feared we would be stuck maintaining this albatross of a code change forever.

Several months later, we were investigating a separate issue in Production and reviewing settings files. This time, with calmer heads and more trust developed between the groups, we had copies of the actual files sent to us by the Ops team. Lo and behold, despite assurances that they had been double-checked, we found a setting that had not been made as specified. It was very subtle and easy to miss — but we had found our culprit.

The lesson here is to trust your teammates, but in the end, the quality of the product should still count more than any risk of hurt feelings. Ask for actual proofs when necessary. Be diplomatic, and be trusting. But verify.

Asking if something was double-checked is not the same as actually double-checking it. If you can make mistakes, so can everybody else. “Teamwork” is a two-way street.

Let me leave you with one final thought. This anecdote was also what first convinced me of the value of DevOps, even before that concept became a marketing term. DevOps is about much more than just bridging gaps between developers and system administrators. It is about process.

Think beyond the creation of a software product to the act of deploying and managing it in production. Then think about that act as something to be defined, managed, and monitored. That is really DevOps in a nutshell.

Tools such as Docker and Terraform have changed the industry in many ways, but none of these tools made something possible, they just made it easier. “DevOps” has eliminated any final excuses we might have had to not automate. Any organization that fails to leverage these concepts today is giving up significant competitive advantage.