Google’s AI Strategy Flaws: An Ex-Googler’s View on the Gemini Failure
or….. TL;DR: Advancement in Google requires an allergy to risk
My background: I spent 6 years at Google, with 18 months in DeepMind (known as Google Brain at the time).
It’s widely acknowledged that Google’s rollout of the Gemini chatbot was rushed, and ultimately unsuccessful. Most articles have focused on failed attempts to be diverse, and I’m surprised no one has dug more into why Google was rushing Gemini to market in the first place. Here’s my take on what went wrong in the months and years before launch.
Googlers had in fact been developing chatbots internally years before ChatGPT came on the scene. I remember testing one internally in 2019. However, productionizing chatbots wasn’t a priority until a competitor scared them into action.
For the past 10 years, except for Google Cloud and the Assistant, almost no new initiatives were considered a critical priority. Notably, both Cloud and Assistant (and Google+ in 2011) sprung up only after competitors established the markets first.
Chatbots fell into the same follow-a-competitor strategy. Leadership was unwilling to dedicate serious resources, oversight or urgency to any meaningful departure from search and cloud businesses until a competitor showed them how valuable the market could be, and their software stack was years behind.
Google top leadership’s modus operandi was to ask each org to develop their own strategy. Org leaders asked lower level leaders to come up with their own roadmaps. This continued down to the individual contributor product managers.
The mindset up the leadership chain is to see what can be achieved with the least headcount possible. Projects requiring fifteen or more engineers at once became rarer and rarer, unless the concept had already gained some traction with users, or more likely Google leadership. This meant it was often impossible to make a sufficient investment in anything high risk enough to actually see the possible return. As a result, across the company you saw dozens of low risk, small-scale projects, mostly completed by the individual teams’ engineers.
Under this system, Google ultimately evolved into a company that staffed and rewarded only incremental improvements on the existing products.
Under Sundar Pichai’s leadership Google has engaged in what Marty Cagan would call “Strategy Theater” in his blog post with Lea Hickman on Transformation Theater.
Similar to discovery theater and Agile theater, most companies today talk about product strategy, but for most of them it is much easier to continue to give allocations to the various stakeholders and just let them decide what they each need. This is sometimes referred to as “peanut butter product strategy” as in spreading the resources very thinly over the company, doing many things but nothing well.
In my circles at Google it was an open secret that Google leaders were bad at strategy and never pretended it was a strength. Coincidentally almost everyone I had these conversations with has since left the company.
What Bad Strategy Looked Like In Practice — My Experience With Google Maps
Individual teams would typically come up with and pitch their own projects. Leaders of the broader orgs (ex: Maps, Ads, Search, Brain) acted more like cheerleaders than actual leaders.
The annual org wide product roadmap was always hard to track down. When I could find it, I always saw it contained a laundry list of “priorities” which were in fact … individual projects. Direction from my immediate and skip level managers was never more than requests for specific projects, or specific bug fixes. On occasion we might get a bigger mandate like “go all in on AI” or “focus on video to compete with TikTok.” Each team would then puzzle through what this meant for their feature.
My team and I were usually happy to work independently, and the lack of top-down orders was seen as a benefit of working at a company like Google. The engineers on my team also had a strong voice, and tended to veto anything requiring significant collaboration from another team or over eight months of work. Long term, this meant we worked on an endless stream of low risk small-scale projects everyone agreed looked good on a promotion docket. Our approach was the norm, not the exception.
I knew it was time for me to leave Google because my manager at the time said I would never be promoted if I couldn’t learn to better “themify” the yearly list of disparate projects.
This was a skillset that would only be useful politically at Google, and it disconnected me from the value our products brought to actual users.
The bottom up approach of short-term impact-focused work meant it was impossible to make any big bets, especially if there was any risk.
One example of this in Maps was a half-hearted investment in social. In 2019 Google launched the ability to “follow” local guides, but the feature was hidden and not widely used. As the PM for categorical search and typeahead suggest, in 2020 and 2021 I pitched my team to build the ability to search for local guides. You needed to be able to look up Gordon Ramsey’s profile if you wanted to see what he recommended in your city.
My engineering team refused to work on this, citing concerns that it would not be widely used after launch. This was a valid concern, given the short term impact goals they were rewarded for in their bi-annual evaluations.
I argued to leadership that full support of social features was needed if we wanted the initiative to have even a fighting chance of seeing adoption. The leads agreed in principle, but no requirement was ever issued for the work to begin. The small team working to make maps social continued developing small, mostly hidden features like before.
Social features might not have been the answer to Maps’s future, but we never committed sufficient focus or resources to properly test it in the market. A more strategic approach would have been to make a handful of large investments, knowing some would fail.
Instead all potentially game-changing projects were effectively doomed with a lack of proper resourcing.
Real resources were only granted once a competitor demonstrated the value of the market (like with Cloud and Assistant) However at this point, lack of investment meant Google was starting late in the game.
The Biggest Miss — Nvidia Ate Google’s Lunch: How Bad Software Crippled a Superior Chip
You’ve probably heard about NVIDIA’s recent rallies due to skyrocketing sales of GPUs for GenAI models? The company now has a higher market cap than Google, even though Google created their own hardware nine years ago specifically to train ML models.
Why did Google fail so abysmally to become a major contender in a domain they led for so long?
A quick review of the timeline of relevant events:
- 2015 — Google deploys the first generation (inference) TPUs internally, and Google’s ML library TensorFlow is released to the public
- 2016 — Facebook’s ML library PyTorch is released to the public
- 2017 — Google invents the transformer model (a critical component of GenAI models, thus driving so many NVIDIA GPU sales)
- 2017 — Google announces the second generation (inference and training) TPU publicly
- 2018 — Google TPUs launched to Google cloud and Sundar Pichai says “A.I. is more important than fire or electricity”
- 2024 — TPUs still have very low adoption after multiple new generations of hardware are released, this is in spite of Google’s arguments that they provide superior performance, especially for transformer models. PyTorch is overwhelmingly preferred to TensorFlow (source)
The cause of Google’s failure is once again their peanut butter product strategy, which tragically was even applied to AI, their supposed lifeblood, resulting in an unwieldy and limited hardware product almost no one wants to use.
These quotes from Hacker News accurately sum up what I’ve heard from others in the industry.
The effort to make (PyTorch) code run on TPUs is not worth it and my lab would rather rent (discounted) Nvidia GPUs than use free TRC credits we have at the moment.
The thing that all of these hardware companies don’t understand is that it is the software that keeps the boys in the yard. If you don’t have something that works as well as CUDA then it doesn’t matter how good your hardware is. The only company that seems to understand this is nVidia, and they are the ones eating everyone’s lunch. The software side is hard, it is expensive, it takes loads of developer hours and real life years to get right, and it is necessary for success.
According to the book Chip Wars, Jensen Huang invested $10B in NVIDIA’s software stack CUDA since it was released in 2006. Je often took a lot of hate from Wall Street for the investment (source). But even though NVIDIA had the head start, CUDA didn’t start supporting neural nets until 2015, the same year Google released TensorFlow. TensorFlow even had a head start on the competing library PyTorch, but a 2023 study showed that only 4% of ML repositories used TensorFlow compared to nearly 70% in PyTorch (source).
Just to clarify, the launch of TPUs was in fact a major investment, and required a lot more than fifteen engineers. They were a rare bet not in the follow-a-competitor playbook. But the resources were focused on enabling TPUs for internal teams to save on Google’s compute costs. Internally, engineers would put up with a lot more to get the job done than an external developer community with more choices.
From 2018 to 2020 I was a product manager working in this domain at Google Brain (now DeepMind, at the time the org where TensorFlow and TPUs lived). I saw firsthand how deploying models on TPUs, even internally, was so difficult that in spite of the cost savings, very few Google teams were up for the challenge. The few teams (with especially large compute bills) that were required to use TPUs suffered through incredibly limited tooling and the cumbersome TensorFlow library.
Due to the poor tooling, adoption of TPUs was much lower than it should have been both externally (via Google Cloud) and internally. The few teams that did adopt TPUs saved Google enough money to justify the project, but it was such a pain to use TPUs that wider adoption from smaller teams and external developers was unlikely.
When I was in Google Brain there was never any urgency to fix this. There were small investments in TensorFlow improvements, but it was nowhere near enough to keep pace with the needs of the industry. There was also heavy resistance to making TPUs more compatible with other libraries like PyTorch (again this would have required much more of a staffing investment than was ever given).
The slow progress was partly due to Google’s promotion criteria. People were rewarded for launching new things (like the subsequent generations of TPUs), and meeting follow-on targets like “reduce latency by 20%.” Growing adoption externally might have also been rewarded, but choosing this as the primary objective took on unnecessary risk. Making a bet on new features might not work out, but a product manager could always count on Google’s talented engineers to hit performance targets.
Leadership also adopted a laissez-faire attitude towards TPU adoption internally. Even though TPUs were objectively much more performant than the alternatives, most teams were allowed to adopt (or not adopt) TPUs at their own pace. This meant minimal adoption, because why would a team waste engineering years getting an unwieldy system to work when your cost center would pay more for CPUs or GPUs?
The goals of Google Brain leadership generally were also much more research oriented, encouraging the creation of new tools worthy of papers, rather than pushing for wider adoption of Google’s tools.
As a result, the relatively limited staff that was tasked to work on TPU tooling was heavily bifurcated. Multiple groups developed similar tools, with a spirit of competition rather than collaboration. While some competition could be healthy, the result here was duplicative work and mediocre results. The problem of duplicative projects led by competing engineering factions has been common at Google for the last decade. This stems from the promotion criteria rewarding new launches above all else (with healthy usage numbers being a nice to have), and leadership being unwilling to take a risk to pick one approach over the other.
The engineers in Google Brain were some of the smartest people I’ve ever worked with, and with better direction from above, and a lot more staff, they might have been able to do a lot more with TensorFlow and TPUs. But that wasn’t how things were done in Google, and especially not in the research oriented Google Brain, which was probably never the right home for a potentially game-changing revenue generating product.
Maybe now that NVIDIA has demonstrated the market potential for performant AI hardware paired with a robust user-friendly software library Google will rethink its investment in TPUs. But the best time to make that investment would have been when Sundar Pichai said “AI was more important than fire or electricity.”
Google had the resources then to become the leader in ML tooling and hardware, had it been a priority. But Google’s risk averse leadership was too afraid to take a stand on what the AI future would look like.
Even with proper resourcing, catching up now will take years, and with NVIDIA’s first-mover advantage Google will have to offer even more compelling reasons for developers to switch.
Will Google think differently about big bets going forward? I suspect not. Google’s 180K machine of impact focused incrementalism has mostly served them (and their stock price) well over the last decade. Jensen Huang faced backlash from Wall Street for years over his investment in CUDA and NVIDIA’s stock price suffered. Furthermore, Google as a company has evolved its people and processes to live and breathe this approach so completely. It’s possible the existing leadership would be incapable of pivoting. After all, except for Thomas Kurian of Cloud, Google’s leaders today advanced into their roles because they did it so well. Those with a higher appetite for risk have long since left the company. Is there anyone left who could actually turn things around?
Ironically, Gemini has done an incredible job helping me edit this article, including suggesting better titles and section headings. Then again, so did ChatGPT.
