Trailblazing in an often under-appreciated role
The experience of shattering a glass ceiling
“Are you crazy? Amazon will never promote an SDET to Principal!” said my friend with a dismissive laughter. The year was 2009. I had just started at the company, and I was eager to grow my career. Was I in a dead-end role?
A glass ceiling is a metaphor used to represent an invisible barrier that prevents a given demographic from rising beyond a certain level in a hierarchy. It was coined to describe obstacles that prevented women from being promoted to top jobs in management. Hats off to those who ignored the precedence of history and relentlessly fought to break their glass ceilings! The term has been broadened since to describe other types, like race-based, religion-based, etc. My “glass ceiling” was not so dramatic or historic. It was subtle — but very personal to me. I was in a software engineering role that for decades was not deemed important enough to ever reach a certain level. Yet I stubbornly got there, against all odds. And it opened the doors for others, after me.
To understand the context, I need to take you back more than 2 decades.
How I got into Engineering Productivity
In the nineties, the software industry at large had Developers (who wrote code) and Testers (who tested the code). Traditionally a Developer role had a higher technical bar and a higher pay, which created an unwritten class system in the industry. There’s still a leftover stigma today.
In 1997, when I graduated with a Bachelors in Computer Science, I had zero interest in being a Tester. I had spent four years in college learning how to write code, so naturally, I wanted to go somewhere and write code. Microsoft offered me a Developer (“SDE”, Software Design Engineer) position. But there was one little problem. I had been randomly assigned to a team and its technology was unappealing to me. It was so boring and uninspiring that I don’t even remember what it was.
The recruiter came up with an alternative. There was a team doing Natural Language Understanding. The prospect of enabling computers to understand human languages inspired me. Microsoft Research in the nineties was cutting edge. I knew I had to be a part of it all. The catch was that the team didn’t currently have an SDE role. It had an SDET role (“Software Design Engineer in Test”). I balked at this. No, I didn’t want to be a Tester. I was well aware of the prejudice in the industry (and I was even perpetuating it with my attitude!). The recruiter explained to me that SDET was a “hybrid” role — not quite a Developer, but not a Tester either. It was essentially a developer that focused on writing test automation, but ultimately, it was a position writing code.
I figured I would rather be an SDET in a team that I cared about than an SDE in a team that I didn’t care about. So I took the job, without knowing a single thing about testing and with zero plans to remain in the role long-term. But over the years, I fell in love with software testing and engineering productivity as a discipline, and I’ve spent 24 years doing this in some way or another, at Microsoft, Amazon and Google.
Yes…a somewhat hasty decision I made when I was 21 shaped my entire professional career.
I liked the role, and found many intellectual challenges that captivated me and kept me in it. But I hated that it was widely assumed that you became an SDET if you weren’t good enough to be an SDE. I knew perfectly well I was good enough to be an SDE: I had graduated first in my class with a degree in Computer Science with a 4.0 GPA and I had been writing code since I was 8 years old. And it was also widely assumed that your impact testing code could simply never be as high as the impact of the person actually writing the code. So, with the prejudice of “lower skills and lower impact,” it was not a huge surprise that the percentage of individuals at Senior level in my role (SDET) was an order of magnitude from the percentage of individuals at Senior level in the SDE role.
The role also suffered brain drain. The individuals that had the chops to become Senior SDETs were spooked by the perceived glass ceiling and slower or uncertain career growth, and switched to SDE, or management roles, where career growth was more well defined. I myself considered that switch many times. But I was having fun doing what I was doing.
Should I continue down this path at Amazon?
In 2009, I joined Amazon, after 11 years at Microsoft. I briefly considered (one more time) that maybe this was a good time to switch roles, since switching companies was effectively a clean slate. I had been pursuing a Senior SDET position for many years at Microsoft unsuccessfully, so I was frustrated.
But I was intrigued and energized by the impossibility of the task.
In 2009, Amazon was a mid-size software company, probably about 3k-4k engineers (it is huge today, easily 20x that). The SDET community was small, maybe 300 or so. There were rumors that there were one or two Senior SDETs, but I never met them. And, of course, nobody beyond Senior level. The entire company had maybe 200 Principals, all of them pure SDEs. So the notion of becoming a Principal SDET was pretty far-fetched… that was two levels beyond any SDET I personally knew at the time!
I had worked out a nice signing package to join Amazon, and the stock skyrocketed. It was just luck, but by 2012, I had tripled my 2009 Microsoft income. That was liberating. Before, I would have been chasing a bonus, and perhaps made very short-term optimizations to get there. Instead, I could focus on the long-term. I had a financial runway, so I didn’t care what score I got for my performance review that quarter. I cared where I was heading 2, 3 years from now. I could take some risks (and fail). I could take the time to learn a lot and think deeply about the space of engineering productivity and what my place in this world was.
In 2012, I was promoted to Senior SDET. Suddenly, the prospect of reaching Principal wasn’t entirely crazy. Even though there was nobody there, it was the next level.
There is no ‘us’ and ‘them’
Part of a “glass ceiling” is realizing that there’s an array of things X, Y, Z, that the main group is doing and the under-represented group might not be. I started looking at some of the things that SDEs were doing that SDETs weren’t traditionally doing, and decided I should do them too.
- SDEs gave frequent technical talks in broad venues. SDETs didn’t. Just jump in the water, I told myself. So I found the most prestigious talk series at Amazon, called “Principals of Amazon Talks” (“POA Talks”), and signed up. The talk series was endorsed by the Principal Engineering community, so I figured if I wanted them to respect me as a peer, I should start now.
- SDEs filed tons of patents on their work. For whatever reason, SDETs did not file patents very often. Our work is just as valuable and innovative, I told myself. I filed an idea, and it got rejected. I filed another idea, and it got rejected. I stubbornly filed a third idea, and it got approved. Over the course of the next decade I would successfully file 35 patents in software testing and engineering productivity topics on behalf of Amazon.
- SDEs interviewed SDEs and SDETs interviewed SDETs. If I wanted SDEs to respect me as an engineer, I should be able to interview SDEs just as well as anybody else, I told myself. So I signed up to become a Bar Raiser (“BR”), which is a specially trained and certified Amazon interviewer that among other things has the responsibility for the final Hire/No Hire decision. Training was intensive and demanding, and I flunked out. I stubbornly tried again, a little later, and after months of training and shadowing I was certified. I conducted 813 interviews in my time at Amazon, and yes, most of them for SDEs.
Could I have Principal impact?
One question remained. Could I, as an SDET, create something that was impactful enough that could justify a promotion to Principal Engineer? That was a tall order. I could see why the path to become a Principal SDE was more well-lit. You’re creating customer-facing features and you can simply point at the revenue these create. But as an SDET, you’re focused on engineering productivity. Your job is to help other engineers around you be more efficient. That’s harder to quantify.
So I did a quick back-of-the-napkin calculation. This math could probably apply to any big software company out there. Assume a cost per engineer of $300k/yr (from levels.fyi). This is conservative: engineers cost the company a lot more than just their salaries; there’s all kinds of overhead like real estate, hardware, managers, HR, etc. Let’s assume engineers spend 15% of their time testing. That comes to 50k per engineer per year on testing. With a company of 30k engineers, this adds up to 1.5 billion dollars per year for the space.
Of course there was opportunity for a testing-focused individual to have huge impact chipping away even a fraction of over a billion dollars of yearly expenses!
I was lucky in that I had created a technology at the right time. I had built a synthetic traffic generator to help a few teams around me do load and performance testing, and I had been generalizing it and expanding the features, mostly for fun, on nights and weekends, as a pet project. There was a series of high profile operational issues with amazon.com that could have been prevented with better load testing. I had the technology! It could run at millions of transactions per second on thousands of machines. I jumped at the opportunity to expand my scope, embarking on an intense journey of relentlessly advertising and growing my platform, company-wide. It was impactful in three different ways. (1) if under-scaled, service owners could use it to prevent bottlenecks that could cause downtime, avoiding profit loss. (2) if over-scaled, service owners could use it to right-size their fleets, saving hardware costs. (3) it significantly reduced the toil of doing load testing, saving engineering effort. Given the size of the company’s operation, these 3 aggregated to many, many, millions of dollars.
I was also lucky in that I had amazing managers. Claire was the manager that convinced me I could, and started my promo doc, both with empathy when I felt down and transparency and relentless firmness when I needed a little push. Dan was the manager that finished my promo doc and got it through the finish line. Cary and Joel were the two Principals around me that gave me encouragement, guidance, and in so many ways were role models to me. All four are still today my dear friends.
The Finish Line
Being a trailblazer was both a blessing and a curse. On one hand, when you’re trying to do something that nobody has done before, you have no clear role models to emulate. Because there’s no frame of reference, it’s unclear when you’re there. You couldn’t stipulate ‘well Chris did X and got promoted, so if I do X I should get promoted too.” On the bright side, I figured, if nobody has done it before, there is no fixed path to follow… I could make my own path!
I was up for promotion to Principal SDET exactly two years after being promoted to Senior Engineer, which was surprisingly fast, even more so for a role that didn’t even have a Principal.
I didn’t get it that first time. But 6 months later, I tried again, and the promotion did go through.
I wasn’t the only one. My friend Pat was also promoted to Principal SDET in 2014, for this work dramatically reducing UI test flakiness company-wide, which also saved millions. The “Principal SDET” glass ceiling had been shattered, not once, but twice. All of a sudden, Amazon had two Principal SDETs!
Because there was no specific Principal SDET promo process, we just went through the standard Principal Engineer promo process, so we had the option to drop the “T” and be Principal SDEs. Pat chose that. I wanted to keep the “T” — I felt it was a badge of honor. It reflected I had reached my position via an unorthodox path. It reflected my love for Testing and Engineering Productivity. And it showed to more junior SDETs around me that the role was in no way a dead-end role. I made it a point to be visible to others — by speaking at conferences, for example.
The Principal Engineering community at Amazon was warm and welcoming. Every single PE had gone through the exact same company-wide centralized promotion process, so I think people trusted I had earned my place in the community as much as anybody else there. Most PEs were curious about a “Principal SDET.” I was a unicorn.
One important aspect of breaking any glass ceiling is what happens afterwards. Do others follow? They did. I was promoted in 2014. In 2017, Phil became the next Principal SDET. In 2019, Mihaela and Ryan joined the club. All 3 were good friends of mine, so we had countless chats about their career growth, and I had the privilege of endorsing all 3 of these promotions. Is there room for a Senior Principal SDET at Amazon? For sure. I’m looking forward to seeing what great promotions come out of the community in the future!
I think at some point we all encounter invisible barriers that prevent us from rising beyond a certain level in a hierarchy. I encourage you to challenge that status quo. Sometimes the path-less-traveled is not well lit, and lonely. But there’s unspoken beauty in it and satisfaction at the end of it.