Geek Culture
Published in

Geek Culture

Belonging to Amazon’s Principal Engineering Community

“The State of the Union is Strong”

Peter Vosshall, one of Amazon’s Distinguished Engineers, kicked off the 3-day offsite with his traditional greeting: “The State of the Union is strong.” Here we were, the three hundred or so Principal Engineers of Amazon, back in late 2014. We had all converged on Suncadia, an upscale, remote mountain-side resort in Eastern Washington. The next 3 days had a loose agenda of talks, open space, and a lot of free time to bond, a significant percentage of that over food, coffee, whiskey and a fire.

Suncadia, remote mountain resort in Eastern Washington and host to our yearly get-togethers

How did I get there?

I worked at Amazon 2009–2020, and was a Principal Engineer (“PE”) for the last six years. I should clarify that Amazon Principal is different from other big company’s. For example, the bar is significantly higher than Microsoft’s, but lower than Google’s. The red line here shows where the Amazon PE promo bar sits in relation to others. It is roughly the top 3% of the engineering population.

The road had been intense. I had worked extremely long hours and sacrificed a lot. I failed my first promo attempt. Much later I learned most PEs failed their first attempt. But at least that first rejection came with crisp and actionable feedback. Almost there; meeting the bar for this and that; do less of x and y; do more of a, b, c. Six months later, my second attempt went through.

Promotions at Amazon were the same for all engineers up to Senior level:

  • Your manager wrote a doc (generally with a lot of help from you).
  • The doc needed 5–7 individuals with levels higher than yours endorsing it (which required a hefty writeup on their part).
  • Then, promotion meetings happened at Director level and included all the Senior Managers and Principal Engineers in the organization, scrutinizing the package. After some spirited discussion and group voting, the Director made a final call.

But Principal-level promos required a few additional steps, including two Assessors who did a dive deep into everything you ever did. They read your design docs and whitepapers, analyzed any code you ever wrote, read what you commented on other people’s code reviews, watched your talks, and physically interviewed people around you. This was an extremely heavyweight process. Once I became a PE, I did assessments myself and they generally took 20-40 hrs of work, spread over a couple of months. The 2 assessors worked independently, weren’t from your team, and weren’t supposed to cross notes, to avoid biasing.

Because this process was expensive, when a manager wanted to put somebody up for PE promo, they needed to submit a request for assessors to a panel that would decide if it was promising enough to warrant assessors. A number of PE promo attempts died at this stage. For the ones that got assessors, if both assessors said No, the promo attempt died at the next stage. If one said Yes and one said No, it was up to the manager’s judgement whether to proceed or not — most cases were pretty doomed with one assessor saying No. You really needed both assessors to say Yes to have a solid case.

At that point, PE promo attempts continued the standard flow of other promotions at Amazon. Write the promo doc, gather the endorsements from peers and partners, defend it in front of your Director and all the directs at the promotion meeting. But for PE promos, they continued their way to a VP-level promotion round and got more scrutiny there.

And then, just when you thought you were in the clear, the scariest part! Even after Director and VP approval, they had one more round of scrutiny at “the Panel.” The Panel was a select group of Distinguished Engineers and VPs from around the company that once again ensured PE promos were following a standard company bar. The Panel could (and often did) veto promotions that were approved by VPs.

Amazon’s fascination with its PEs

Promo attempts for Principal could die at any one of these steps. At Suncadia, Peter calmly went over the funnel that year at the State of the Union — how many people had requested assessors, how many had their promo attempts approved by their orgs, and how many the Panel had accepted. “New Principals this cycle, please stand up!” Peter said with a smile.

I stood up, blushing. It was a powerful moment. I realized just how much collective effort had been given to get me to this room that day. I had worked hard, but also, the company had invested significant resources to help me grow, and subsequently ensure that I met a bar. So many people had spent so much time looking at every aspect of my professional life to decide if I belonged there. I felt a huge amount of responsibility on my shoulders. I was expected to be a role model. I was expected to exhibit composure and high judgement. Dozens of people had scrutinized every aspect of my work and my personality. I was going to be trusted with decisions that could impact the lives of millions. I was excited and terrified at the same time.

And hereby lies Amazon’s fascination with its Principal Engineers.

Amazon does a huge amount of due diligence before they promote their engineers to PE because PEs have a ridiculous amount of power.

Google, where I’ve been working for the last year and a half, aims at removing bias by having committees (i.e. hiring, promoting) where decisions require consensus from multiple people for various parts of the company. Amazon has a different philosophy: ensure specific senior individuals have a proven track record of high judgement then give them inordinate amounts of power. Each approach has pros and cons, but they do sit on opposite ends of the spectrum.

Let me give you an example to contrast the Amazon and Google philosophies: making a hire decision. I did 800 interviews as a Bar Raiser (“BR”) at Amazon. BRs are interviewers that are single-handedly responsible for making the final hire/no-hire call during the debrief. That’s a huge amount of power to place on just one person. But the BR certification process was brutal, with me having to shadow and being shadowed for dozens and dozens of interviews, track metrics, with mentors scrutinizing and critiquing every question I asked and every minute I spent on that interview and eventually signing off that I met the bar to be a BR. Train and assess the heck out of somebody, then if they pass, enable them to drive. Google focuses on removing bias and offering an experience that is inclusive and fair to all (which I very much like), so there’s a group of people who interview you and a group of people who (together) make a hire/no-hire decision, based on the interview feedback from the first group. I have mixed feelings: I love Google’s commitment to fairness above all, but my Amazon reptilian brain wonders if you can achieve that with people who have demonstrated extremely high judgement consistently.

Now that I was a Principal, I realized very quickly that people would listen to what I said like it was the gospel. I got in the habit of speaking last during a meeting, as to not shut down opinions of more junior engineers that could be intimidated by the level. Great Amazon senior leaders always spoke last for that reason. This may be counter-intuitive, but the more senior you are, the less you speak and the more you listen. I’ve been in a number of meetings with Andy Jassy (Amazon’s CEO) and he is quiet for the majority of the meeting, thoughtfully listening, asks a few questions here and there, and only states an opinion after everybody else has spoken.

To combat the curse of the level, one of my favorite Principals perfectioned the art of asking questions instead of stating opinions. The questions he asked were clearly aimed at helping you reach a conclusion he had reached a long time ago, but rather than telling you that conclusion, he always asked the right question so that you could get there yourself. Instead of being told what to do, you reasoned through it.

With seniority comes this nugget of wisdom: More listening, less speaking. If you must speak, more questions, less opinions.

Facebook combats the curse of the level by making levels private. Everybody is a “Software Engineer” and you don’t know if the person you’re talking to is a junior person or a senior person. The idea is that what you say should be considered without bias as to your level.

I like that egalitarian view, but I have concerns too. Back when I worked at Microsoft in the nineties, our levels weren’t disclosed either. But circa 2002, Microsoft changed that. I remember what a treat it was to wake up that morning and look up what level all my co-workers were!!! We immediately started gossiping over chat. “Can you believe that moron is level 65? He doesn’t do anything!” or “Can you believe that hard-working person is only level 62? I thought they were a 64!” In the end, and over time, I believe it drove accountability, or at the very least it shined spotlights on the cases where the person’s performance was clearly unaligned with their level.

At Google (my home now) people can opt to show their levels or hide them. Ultimately, I believe in accountability and transparency, so I chose to display it.

The Principal Community in 2014

Having a single PE Community was extremely important for Amazon for a number of reasons. It was connecting tissue that miraculously held incongruous parts of the company together. For the most part, Amazon operates like a chaotic collection of startups with their own agenda, yet it somehow behaves cohesively. PEs are the glue that enables this. When your team and another team were at odds and couldn’t work things out, the Principals from both orgs would grab coffee and calmly work out boundaries, compromises, and a solution that was best. Bonding over meals, coffee, vast amounts of alcohol, or simply sharing a bonfire in a remote mountain resort for a few nights built friendships and relationships that you leveraged constantly to advance your ideas and help your team. I made it a point that first Principal Offsite to meet and greet every single Principal around me.

But, Amazon’s growth was a curse for scaling the PE community. In 2014, when I went through the process for the first time, with 10k engineers and 300 PEs, we were still small enough we could know each other. For a while, the company was doubling in size every year (and so was the PE community). In 2021, there’s 70k engineers, so probably 2000–2500 PEs. I can’t imagine the same dynamics that existed when we were 300 working with 2500.

The PE community did manage to stay together for a surprisingly long amount of time. I think it was the long-tenured people like Peter (who just retired after 22 years at Amazon) that held it together.

We kept attempting to meet every year. Suncadia was a massive resort, but our PE Community outgrew it. In 2014 and 2015 we rented part of it; in 2016, the entire resort; in 2017 anything else around it. In 2018 we sharded the community: the first half on Mon-Wed, while the other half on Wed-Fri, with Wednesday being a day where briefly we were all together again. In 2019, we were about 1200 so we rented an entire conference center at the Sheraton downtown Seattle. But the small community vibe was gone. I think that marked the end of an era for the company — and for me.

We had weekly lunches (Amazon was very Seattle-centric; it has since become more global). You’d bring up your food to a conference room and engage in some spirited debate with your peer Principals for an hour. Most of the PEs were in the headquarters but a few joined by Chime from other parts of the world.

We hosted a company-wide weekly technical talk series called the Principals of Amazon (“POA”) Talks. These had a large audience of maybe ~1000 people (in-room and connecting remotely). Anybody who wanted to speak at a POA received a couple of opinionated PEs as coaches who gave endless feedback over endless dry-runs, and had to eventually approve it as worthy of the POA brand before they went up on stage. Giving one of these talks was a badge of honor (I gave one in 2012 and it was exhausting!).

Giving my Principals of Amazon (POA) talk in December 2012. “How we tripled our load in production for testing purposes and lived to tell the story”

We also provided a company-wide Design Review Process. Anybody in the company could submit a request, explain what they were building, what kind of expertise they were seeking, and all of us would get a notification and any of us could sign up to help out if it matched our skillset. Having your design review done at the Principal level was too a badge of honor. Principal Design Reviews were key to operating as a cohesive company even though we were hundreds of little startups under the hood. We connected people trying to build similar things. We injected company-wide best practices. And the breath of opinions from around the company always enriched the discussion.

Participating in all these company-wide initiatives was a 2-way street. I helped others, but also, it helped me. I became a better engineer simply by being exposed to so many amazing people and their wealth of knowledge and viewpoints. I figured every exposure was an opportunity to learn from different role models from around the company.

I think as companies grow, these communities fade away, simply because the scaling mechanisms just no longer work. You can apply sharding to have smaller autonomous groups and then apply aggregation processes to bring back ideas from all of them. Maybe you have communities per VP, or some other affinity. But it’s just not the same. Belonging to Amazon’s PE community was a unique opportunity to get exposed to all hidden corners of a company just as it hit explosive exponential growth.

--

--

--

A new tech publication by Start it up (https://medium.com/swlh).

Recommended from Medium

Supper Club Musings: Nepotism

Learn on the move

What Does Referral Source in Job Application Mean?

Learn Anything: Are online courses worth it? [3]

My Professional Organizing Principle? Country Transformations!

With all these specialists — isn’t there a need for generalists?

7 Tips for ITSM Consultants

#DayNotes — Season 2019 — Episode 14

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Carlos Arguelles

Carlos Arguelles

Hi! I'm a Senior Staff Engineer at Google. Prior to Google, I spent 11+ years at Amazon. And prior to Amazon, I spent 11+ years at Microsoft.

More from Medium

My Amazon Reptilian Brain

Do Google’s Engineering Practices Work for a Startup?

Rows of books arranged by color on a bookshelf.

Why am I joining Volvo?

CTO interview: Charity Majors, the amazing ops engineer who founded Honeycomb