Ask the Tech Lead: I have to make a technical decision but I can’t know the right answer

I’m hoping to make this a series of posts, discussing the unwritten advice for excelling in highly technical leadership. In the spirit of Camille Fournier’s excellent series, I’ll tackle some of the hardest questions I’ve heard from coworkers and mentees in my time as a lead engineer for teams and groups at Yelp. Huge credit to Jonathan Maltz for help refining this particular post and the overall format.

Engineer debates choosing AWS vs GCP (via Justin Luebke on Unsplash)

The question: How do I make an impossible technical decision wisely?

What’s the right approach when I need to make a big bet but have little guarantee of the right call in the end?

The Solution: Think like a scientist

  • The decision is often between the status quo and some option we have no or little experience in.
  • Sometimes the best and worst case outcomes are either unknown or have a scary amount of variance.
  • Sufficiently new or different ideas often imply unknown challenges lurking. What if those unknown challenges are really bad and make it unappealing in hindsight?
  • New ideas might imply major architecture changes that are outright dangerous. If we fail to control risk, it may not matter how good a choice we make.

The reality is you still have to make impossible decisions, even if the right choice can’t be known until later.

My advice is to work your decision like you would a scientific experiment: deeply invest in learning about the problem, build a hypothesis about the right call, and then, most importantly, propose a series of small, incremental experiments to build confidence that your choice is correct.

As a whole, my approach to these big questions looks like:

  1. Frame the big question and take an opinionated stance on the answer based on whatever data is currently available.
  2. Come up with an initial experiment to partially vet that stance. It should be quick (~1 quarter) to accomplish, give meaningful directional feedback on whether the opinionated stance is still correct, and hopefully provide engineering leverage to test the next experiment more easily/quickly/safely.
  3. Evaluate the experiment’s result. Does it suggest our opinionated answer to the original big question was right? Wrong? Have you learned something that needs to alter your answer to the big question?
  4. If needed, update our best current proposal from this feedback.
  5. Rinse and repeat steps 2–4 until you’ve answered the big question empirically.

Applying this to a concrete problem

Frame the big question, take a stance

After a little research and my anecdotal survey of industry, I’m sufficiently curious about the alleged development velocity and ease-of-use of Lambdas to take an opinionated stance: “we should use Lambdas by default with Step Functions, with the pull-based architecture only used as a fallback in rare cases”.

Come up with an initial experiment

So for our first experiment, I aimed small and replaced a trivial task with a Lambda: all it did was reformat some JSON and log the result. This is incredibly “boring” from a technical perspective, but still required making some important directional choices. Namely:

  • How do we deploy and monitor lambdas?
  • Should we use any frameworks?

This line of thought led us to discover a few important hypotheses I wanted to prove/disprove with our first experiment:

  • Is the serverless framework a good tool to leverage to build Lambdas?
  • Can we build a CI/CD pipeline that feels like best-in-class service tooling that actually deploys Lambdas instead of our normal SOA setup?
  • Are Lambdas performant in production?
  • In the end, will this change let us get a functional change through the coding lifecycle faster, without sacrificing safety or architectural sanity?

Evaluate the experiment’s result

Our hypotheses happened to mostly be answering yes/no questions, but this isn’t the only way to measure success. Depending on your particular experiment you may want to check business metrics or even squishier human ideas like “oncall happiness”. The important thing is that you have a clear idea of what you want to learn from your experiment and an clear sense of how you plan to measure that learning once the experiment is live.

Update our overall proposal and iterate

This feedback loop (experiment, see results, modify hypothesis, experiment again) is crucial to noticing problems and course correcting. Make sure that even your big nebulous bets have a clear way to learn + iterate and you’re comfortable with the cost of the worst-case outcome. This helps protect your project from losing momentum halfway due to an unsuccessful experiment.

Final thoughts

  • What’s the big idea? (add historical context, frame the opportunity)
  • Who are the owners this project(s)? (clear ownership helps avoid death by analysis and offers clear points of contact)
  • What’s my best guess at the right answer? (doesn’t have to be correct in hindsight, just has to be a useful, opinionated stance that implies action)
  • What experiments have we already done in this effort? What did we learn from them?
  • What is the next, most valuable experiment we should try?

The ability to take on a vague project of potentially massive scope without succumbing to a blind guesswork is quite challenging. But on the upside, it’s also a rare skill that you’ll use more and more as you become a more senior technical leader.

Luckily for all of us, the key isn’t to magically know the right answer up front. Instead you need to build deep context in the problem space — really live and breathe it — until an initial educated guess forms itself. Your first experiment should provide directional feedback and force you to take theoretical ideas and build them for real. Then each ensuing bet shapes your hypotheses about the overall question, and guides you down the rest of the decision tree for your project.

engineer @yelp, public speaker, lover of affogato

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store