Aug 31, 2021
The Paradigm Shifts with Different Dev:Test Ratios
How things changed going from 1:1, to 10:1, to 100:1
I’ve been involved in various aspects of engineering and operational excellence in three large companies (Microsoft 1997–2009, Amazon 2009–2020, Google 2020-now). In those decades, I’ve seen three major paradigms in the relationship between developers and test engineers, depending on the dev:test ratio. Each paradigm shift forced me to evolve and think differently about how to be effective at my job. It both threatened the future of my role and offered new growth opportunities.
It all came down to leverage and force multiplication. My favorite analogy is pulleys, which change the magnitude of a force. With a single pulley, you have to put 10kg of force to lift 10kg — that’s your 1:1 dev:test ratio. With a double pulley, you need less force — that’s your 1:10 ratio. You can make increasingly complex pulley systems that allow you to lift 10kg with less and less force — that’s your 1:100 ratio.
First, some background. Microsoft had SDEs (developers), STEs (testers), and SDETs (Software Design Engineer in Test, a hybrid between a developer and a tester — it is a developer position, but you focus on creating automation for testing purposes and developer productivity). I started my career as an SDET. Amazon followed that recipe with SDE, SDET and QAE (testers). Google was pretty similar, with SWE (developer), TE (tester) and SETI (the hybrid role). Each one of these roles had its own idiosyncrasies at each company, but there were enough similarities.
Phase I — The 1:1 dev:test ratio days
When I started at Microsoft in 1997, we had a 1:1 dev:test ratio, meaning there was as many testers as developers. I think one of the motivators was the static, waterfall nature of shrink-wrap software. Back then, we were certifying builds of things like Microsoft Office, or Microsoft Windows. The product was released roughly every three years. And when it was released, it meant burning 50 million copies of the CD, and physically shipping them to stores. The Group Manager had all the testers sign their approval of the final build in an actual piece of paper that she then framed and hung. I was responsible for copying the final official bits of my DLL onto the Microsoft Office shared drive that was going to be used to burn the master CDs. If you had a bug, that bug would live forever, burned onto that CD for eternity (or until that person used that CD as a coaster for their drink). It wasn’t until years later than Microsoft added the ability for software to ping the internet for updates and automatically install them. In 1997 some customers would maybe download patches manually, if we were lucky. So the stakes were high, and Microsoft mitigated that with a 1:1 dev:test ratio.
This created a number of fairly toxic dynamics though. For starters, the developers didn’t bother to do very much testing at all, because they knew there was a plethora of testers as a safety net. Just throw it over the wall and there’s a sea of testers eager to try out your code and find bugs. I would get code that didn’t even compile, all the time. Developers weren’t the only ones at fault though. The concept of throwing it over the wall worked both ways, and testers would throw bugs over the wall, sometimes without good repro steps, and sometimes not particularly high quality. We weren’t partners.
The 1:1 ratio also limited growth opportunities. It was extremely difficult to achieve a big impact when your job was literally “test the code that Bob wrote yesterday.” I also think that lead to a caste system where testers were second-class citizens. I personally never viewed myself like one: I had a Computer Science degree just like any SDE around me, and I had cleared the SDE technical interview bar when I got hired… I just had an extra “T” in my title to reflect that I found Testing interesting. But there were some that viewed me as a second-class citizen, and here’s a little story to prove it. I went on to create the load and performance testing platform that Amazon uses internally to test thousands of services (here’s that story). Yet when I started writing it, there was a developer in the team that said, “Uhmmm, this is complex code, I’d prefer a developer writes this code and not a tester.” I think I proved that individual wrong. The code I wrote runs millions of times per second against business critical systems all over Amazon, has done so for a decade, and was scrutinized by a number of Principal Engineers for my promotion to Principal.
In terms of my personal career growth, this was an extremely frustrating time. Out of the 24 years I’ve spent as a professional in the software industry, at least 10 of those were unhappily stuck as a mid-level engineer with no idea how to get past that bump and into a Senior role. The upcoming shift in dev:test demographics would provide just the thing I needed to get out of my rut.
Phase II — The 10:1 dev:test ratio days
When I started at Amazon in 2009, we had a 10:1 dev:test ratio. This was a significant paradigm shift for me. The main difference is we were working on web services and not shrink-wrapped software. You could have good telemetry in your web service. Those real-time metrics (CPU usage, JVM metrics, latency of transactions, error rate, etc) were connected to alarms which were connected to pagers, and there was always somebody oncall. If a metric exceeded the threshold, you would get paged, drop whatever you were doing, rollback the environment (these rollbacks could even happen automatically based on alarms), investigate, fix the bug, push the fix, and your millions of customers would immediately get it. It was an entirely different world from shrink-wrapped software where there were millions of copies of your software in the wild burned to CDs or DVDs. With web services, there was a single authoritative copy of your software, in the servers you owned. So knowing when something was broken it, and addressing the problem, were significantly easier than in the old world. Not particularly good for life-work balance, but great telemetry and a pager could make up for a multitude of deficiencies in testing.
In 2009, there were about 300 SDETs and 3000 SDEs at Amazon. The 10:1 dev:test ratio changed so many dynamics. Developers did not assume their code was going to be tested, since testers were stretched thin. So they did a lot of their own testing — not throwing over the wall. Also, if they pushed a bug, they would get paged, and nobody enjoys getting paged in the middle of their family dinner or deep in their sleep, so there was a built-in incentive to test your code better. This was stated explicitly in “Ownership,” one of the Amazon Leadership Principles (Amazon’s DNA codified). Service ownership at Amazon was extremely strong. Developers and testers were in this together, and that led to much stronger partnerships than I saw in Microsoft’s shrink wrapped software world. Part of that was speed of delivery: you needed your code in production in hours, not years, so having a human gate was neither sustainable nor scalable.
But this also threatened the very existence of my role. All of a sudden, we needed one tenth of the SDETs we needed before. Was I a dinosaur? Was my role heading towards extinction?
Maybe. But it also gave me unique opportunities to lead. Overnight, I expanded my thinking from “test the code that Bob wrote yesterday” to “test the code my ten developers wrote yesterday.” I couldn’t single-handedly accomplish all the testing, but I was the expert in testing in the team so others depended on me to dictate what needed to get tested, how, and in which priority order, so that they could tackle whatever tasks I didn’t have the time for. In every team I went to, I built partnerships, instead of having the somewhat antagonistic dynamics we had back at Microsoft in the nineties. Maybe some of that was that I was more mature, but I do believe the ratios and the resulting culture had a lot to do too.
To me, this newly discovered leverage was a golden egg. I learned how to lead others, and how to accomplish more with less. After being stuck as SDET-II for a frustrating decade, I was able to get promoted to Senior SDET in a couple of years at Amazon.
Phase III — The 100:1 dev:test ratio days
There was a small change brewing at these big companies. Microsoft removed the SDET title altogether circa 2015. Google removed the SETI title as well (in reality SETIs didn’t quite go away, but they became developers more officially, in the discipline of Engineering Productivity, or “EngProd”). Amazon meanwhile did not drop its SDET role but slowed down the hiring of SDETs. In 2009, Amazon had 300 SDETs and 3,000 SDEs; in 2020, Amazon had 600 SDETs and 50,000 SDEs, effectively changing the dev:test ratio from 10:1 to 100:1 over the course of a decade.
Again I faced the same dilemma. Was I a dinosaur? Was my role heading towards extinction?
I’m a glass-half-full kind of guy. Others saw doom, I saw a golden opportunity. Engineers needed to test: nobody wants to be paged in the middle of the night. But they didn’t naturally start a brand new job out of school with the knowledge and intuition on how to do it. If you get a degree in Computer Science, you mostly learn how to write code, not how to test code. We hire great, smart engineers, and we expect them to test their code but we do not give them the knowledge or tools to do so. We set them up for failure. We needed to fix that.
I did two things.
One, I focused on building highly scalable tools. Rather than writing tests (like in the 1:1 days), or writing bespoke tools for my team to test (like in the 10:1 days), I focused on writing generic tools that all of Amazon could use. I ended up creating the platform that Amazon uses for load and performance testing (this is that story), and creating tooling that changed code coverage at Amazon (this is that story). These tools quickly got adopted by hundreds, then thousands, of internal developers, and saved Amazon millions of dollars in developer toil.
Two, I focused on building scalable and sustainable education around test practices. We had tens of thousands of developers that wanted to learn how to test. We had a few hundred SDETs and QAEs that were test experts. Could we leverage these test experts to teach the rest of the company how to think about testing? This was a win:win situation. It created growth opportunities for those in test-related roles, to influence significantly beyond their team, and it provided the developers the knowledge they needed to do their own testing. We created courses, gave hundreds of sessions to thousands of developers, and hosted conferences also attended by thousands (this is that story).
In terms of career growth, this super-charged my career. I had just been promoted to Senior, and 2.5 years later I was promoted to Principal. That was an extremely fast promotion, and unusual: for a couple of years I was the only Principal SDET at Amazon. Many told me Amazon would never promote an SDET to Principal. When I was up for promo, there was no Principal SDET process. I just went thru the Principal SDE promotion process, so I technically had the option to just be an SDE (another guy had done that). I refused, and kept the “T” until the day I left Amazon, as a badge of honor. I am proud that I chose the less travelled road and broke a glass ceiling for my role. Three years later, my good friend Phil joined the Principal SDET ranks; the following year, my friends Mihaela and Ryan also joined. Even for Senior SDETs, I saw huge progress: a decade ago, 1% of SDETs were at Senior level; today, 10% of SDETs are at Senior level.
This speaks volumes of how being highly leveraged forced us to evolve, in a positive manner. Sometimes a threat can also be an opportunity.
Phase IV — ???
I don’t know what the future holds. I continue to ask myself on a regular basis: am I a dinosaur? Do I need to evolve?
Here’s my personal take.
I want developers to be strong owners of their product: that includes testing and operations. Being able to punt these to others feels like a lack of ownership. Amazon has this right. When you have to own operations, you’ll care about testing. And by owning development, testing and operations, you own your destiny in full. Isn’t that a better outcome? Why wouldn’t you want to fully own your destiny?
At the same time, we will always need specific people to have deep knowledge in engineering and operational excellence. This can be a developer, a tester, or any other actual role. Not every person will be an expert in testing, so those who are experts and passionate about testing will have 3 responsibilities: [a] consulting, oscillating between many teams, auditing test processes, priorities and approaches, [b] educating others on how to test, what to test, when to test, what techniques to use to test, etc, particularly via scalable methods like courses and talks at conferences, and [c] create engineering productivity tools to reduce or remove the toil associated with testing.
To be fair: there are and always will be specific domains that require a higher number of testers, either because automation is too expensive or because the stakes are too high. And there’s always value to exploratory testing. But time and time again, a high number of Engineering Productivity engineers can often lead to a proliferation of bespoke tools: most teams think they’re special, and they’re not — they can and should use standard tools.
Leverage and force multiplication are the way to evolve. I am certain Amazon, Microsoft and Google each spend over a billion (with a “b”) dollars per year in testing. There’s clearly opportunities there for individuals with a passion for testing and quality to improve this. If there’s a gap, and you have the passion, step in and fill it, regardless of your assigned role. You might be surprised where that leads you.
