Nerd For Tech
Published in

Nerd For Tech

You got peanut butter in my chocolate!

The story of marrying a Profiler and a Load Generator at Amazon

An idea written on a napkin…

Every year Amazon used to treat its Principal Engineers to 3 days of pampering and networking at a resort nestled in the mountains, about two hours east of our Seattle headquarters. We would pause whatever we were doing, drive over the mountain passes, and spend three days getting to know each other over coffee, whiskey, good food and activities. Today Amazon has thousands of Principals so they can’t do this anymore, but in 2017 we were about 500 or so, so we all knew each other.

A sample flamegraph (credit)

A long wait…

The next morning, we all headed back to Seattle. It was only 2 days later I found that napkin in my pocket as I was about to wash my jeans and remembered our long brainstorming session, so I brought it up with the team responsible for TPSGenerator (by then, I had taken a step back from day-to-day management of the product, in a deliberate effort to create space for the next generation of leaders to step up and own the product). They were excited by the idea, and it went in the backlog.

The week that changed everything…

Fast forward 18 months… Most critical-path services on the amazon.com side of the business tend to do load and performance testing to ensure they can withstand peak traffic and they are appropriately scaled at least twice a year, once before Amazon Prime Day (in June) and once before Black Friday and Cybermonday (in November). To build org-wide buzz and excitement, we had a specific week (“Performance Test Week”) when we asked all teams to do this. This also built some amount of peer pressure: all the cool teams around you were load testing that week so you should too! David Treadwell, SVP of eCommerce, tasked Priyanka with the complex cat-herding logistics, so she and I grabbed coffee to brainstorm ideas. I was in AWS, which is an entirely different business unit from amazon.com, but I was always opinionated about load and performance testing so I made an effort to stick my nose in initiatives like these anywhere in the company.

We filed a patent with some of our ideas!

Some final thoughts

  • Don’t just wait for work to be assigned to you: be opinionated about what your product should do, be an Owner, have a seat at the table in making those things happen. Your level doesn’t matter. Your title doesn’t matter. The amount of time you’ve been at the company doesn’t matter. You can be a Leader regardless of those superficial indicators: being a leader is about behavior. Every single one of the volunteers who built the Edmonds Community Center was a Leader.
  • Leadership involves relentlessly talking to people around you, not just being a dark corner writing code by yourself. Often, you need to align multiple teams behind an initiative, rather than work in isolation.
  • Sometimes you have a great idea but it’s just not the right time, for any number of reasons. It took two years for what Tim and I wrote on a napkin during dinner to become a reality.
  • Have your eyes open to recognize when the right time has come, so that you can seize the opportunity then.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Carlos Arguelles

Hi! I'm a Senior Staff Engineer at Google. Prior to Google, I was a Principal Engineer at Amazon for 11 yrs, and before that, I spent 11 years at Microsoft.