Tales of a First Time Tech Lead (part 2): Principles
In the first post of this series, I described in broad strokes the dizzying personal journey that got me to what is, for all intents and purposes, a lead engineering role in a billion-dollar company. There is, however, little glamour to this. It has been a tough ride, equal parts draining and energizing, where every weekday you live by your calendar and where, to quote boxing giant Mike Tyson, “if you’re not humble, life will visit humbleness upon you”.
My intention in telling my personal story is literally to write what I would have like to have read myself some time ago. More than that, I want to give a more human account of this real-life narrative. Although there are countless incredibly insightful articles out there on the different aspects of tech leadership — whose authors I am certainly indebted to — I feel a certain lack of identity to most of them: a lot of you should do this and that but rarely I did this and that…and here’s how it turned out. While I obviously won’t go into internal details about the company I work for, I certainly have no qualms about detailing my own thinking or how I felt about something at some point in time.
I ended part one by alluding to a set of three core engineering principles that have been my guiding posts. Before we dive into those, a word about how important it is for me to have such a foundation is in order.
One of the main reasons why I personally find it fascinating to work in technology is how it is by definition an enabler of something else. Unless you are a pure technology company — selling tech to other businesses — you are in fact working to provide tools and means for a different, more tangible product to find its way to someone, somewhere. In my case, I am particularly fortunate to be working in technology that enables change in the way people eat. If we want to go for the moonshot and be somewhat bleary eyed about it, we can say we work with technology to solve food. Hard to be more ambitious than that, short of, you know, being Elon Musk or something.
The reason I mention this is because I am not allowed to ever lose sight of this fact at the expense of not being able, as a tech lead, to make the best decisions I can on behalf of the customer and, in turn, of the business I represent. In other words, technology is nothing if only for technology’s sake.
As such, it was important for me to try to get as close as possible to the main concepts that, from an engineering perspective, maximize the competitiveness of the company. This is arguably the difference between the IT department being seen mostly as a liability (“takes too long, costs too much, has too many bugs”) or being an actual asset in an aggressive, fast-moving market (“they delivered quality, on time”, “website is rarely down”, “it just works”). At the end of the day, it’s not just about you wanting to be a leader instead of a follower — it’s whether you can afford to be a leader or have to settle for being a follower. I believe engineering, where the rubber meets the road, plays a crucial role in that. But maybe I’m biased, right?
I believe we should be sponges in absorbing all kinds of influences, yet be critical about what we consume and don’t take anything as gospel. In technology, it is very easy to get hyped about way too many things (hype driven development is a thing) but there is nothing wrong with standing on the shoulders of giants. I believe it is in the remixing of the right concepts in a way that fits our specific purpose that we are able to find virtue and even originality. That’s what I tried to do, for the most part, in coming up with my three engineering principles, always having the goals of the business in mind.
Design For Change
Anyone who has worked with a growing (usually monolithic) application for a significant amount of time knows that software becomes increasingly difficult to maintain and extend. There are many reasons for that but to me it boils down to the fact that, more often than not, it is not designed to change. Most of the time this happens due to a lack of any design at all — it just grows, with different people and different teams messing with it to the best of their capabilities. We need to be fast and corners need to be cut.
This is perfectly fine for an early stage startup aggressively looking for market fit. However, as the codebase grows and said startup starts maturing (after hopefully getting that foot in the door) things slowly get to a point where the cost of change just gets too high, both in terms of development time and how difficult it becomes not to break other stuff along the way. You become a full-time firefighter instead of the builder you thought you were.
The reality, though, is that the world is moving at an increasingly faster pace and change is all but inevitable. You constantly have to develop new features to capture market opportunities, you need to quickly bridge the gap to your competitor on that latest feature they released, sometimes you even need to significantly pivot your business to a new model. Slow but steady just doesn’t really cut it.
Techniques like Domain-Driven Design help in taming complexity, develop a common language between technical people and non-technical stakeholders and, although the learning curve for the organization is certainly very significant, it is an opportunity for your team’s code to never drift much from the existing reality of your business. In every decision we make, the inevitability of change must always have a front row seat.
I’ll certain touch on this in more depth later on, but it’s important to note here that, beyond all the hype, a microservices architecture not only forces you to define and better understand your business domains, it helps change by breaking down complexity in smaller, independent pieces.
Design For Failure
It’s inevitable to get lured by the million stories of technology greatness by the Ubers and Facebooks and Twitters and Spotifys of the world, right? They always have it all figured out, nothing but geniuses there, they crack problems for breakfast that we can only dream of maybe solving in a couple of years, maybe never. I am obviously exaggerating but I was happy to see Uber’s Senior Staff Engineer Matt Ranney being very candid to his audiences at the beginning of his talks around the world and admitting that, contrary to every other speaker, they didn’t have it all entirely together. And why?
Because things break.
The idea that failure is inevitable and that we should design with it in mind really resonated with me. Ranney has talked extensively about it and we all know about Netflix’s Simian Army and Chaos Monkey. For me it boils down to changing the mindset of having exclusively the happy path in mind.
Failure will happen so we must get good at predicting every way it can happen and design for that. Be it making judicious use of circuit breakers or thinking your architecture with redundancy in mind, identifying and addressing single points of failure, if you can voluntarily shoot down any single piece of your production environment and still be running, you’re in (very) good shape. That will never be possible if you don’t design every single piece of your puzzle with failure in mind. At the end of the day, compound interest kicks in and the longer you stay up, the less money you bleed. Competitive advantage again.
If I had a dollar for every time someone or some company claims to be “data-driven”, I’d be rich. But reality is seldom like that.
Going back to the idea of the competitive product market, the pattern of making most decisions based on data instead of gut feeling emerges. I will write about purpose in a future post but for now I would just mention that I find it extremely important for engineers to have the means to understand why they are working on something and what the impact of that work is. None of this is tangible without numbers. And numbers, when correctly looked at, give us insights.
Tracking everything is obviously an ambitious — and arguably utopian — goal but, once again, it is about a certain mindset and fostering good habits instead of bad. I believe the habit of measuring increases our understanding, keeps us honest by allowing us to recognize mistakes and regressions, and also makes us much more objective at what we do. Business metrics, application metrics, server instance metrics, team metrics, individual metrics — it can get too much, too fast — but to paraphrase Etsy on their original blog post about StatsD, “if it moves, we track it”.
I do believe it is the hallmark of a high-functioning engineering team to be passionate about data, to understand their systems, services or applications, to get carried away by the excitement of driving performance numbers up or the size of a bug backlog down.
Although I brought forward these three core principles to my team the very first time we got together, applying them is an ongoing battle. It is the name of our game to have to deal with lots of constraints that pull us in different directions and, more often than we would like, we have trouble changing some part of the system, or things break left and right, or there’s something we don’t have eyes on (and we should have) because we hadn’t yet thought to track it.
But then again, Rome wasn’t built in a day, now was it?
If you got this far, thank you so much for reading and I hope it has been useful for you in some way. I’d love to hear your comments below.