Google’s Failed AI Experiment: 3 Key Lessons for Product Managers
Two months back, the tech world buzzed with anticipation as Google unveiled the “Search Generative Experience.” This feature is available for a limited number of users through Google Labs. The pitch? An AI-crafted summary along with the regular Google search results. The promise? A complete revamp of how we seek information and shop online. But here’s the twist: reality didn’t quite meet the hype.”
There is little point in criticizing a product feature clearly labeled as experimental, especially when pioneering new technology. Instead, this article discusses what we can learn as product managers and business leaders when building products powered by generative AI.
Unpredictable Behavior
Among the most pressing concerns tied to generative AI is its potential to spread misinformation. To put this concern to the test, my initial trial involved searching for “US Presidential Election 2024.” The intent was to see what Google would consider an appropriate overview of the current status. To my disappointment, the results looked like any other web search. This is one of the terms for which Google opts not to display AI-driven results.
While testing, I saw this happening often and unpredictably. When building AI products, it is vital to be transparent and manage your user’s expectations when the system doesn’t work as expected. In this example, I would have liked to receive a short, high-level explanation to understand the reason behind the lack of generative output.
User Trust & Misinformation
Admittedly, my previous search term was very open-ended and lacked a clear intent. To sidestep these potential pitfalls, I narrowed my focus to the top Democratic candidates. The results did present an AI-generated response — a roster of names accompanied by a handful of articles, presumably the sources shaping this list. However, the origin of each name remained unclear.
One can only hope that fact-checking will be easier when Google officially launches this feature. Speaking of which, is Bernie Sanders throwing his hat in the ring once more? According to Google’s top results, he’s not!
A critical thing to remember when designing AI-powered experiences is the importance of nurturing confidence in a statistical model’s outputs. Establishing trust is even more important when building business applications.
(Not so) Generative AI
Let’s move on and test how Google’s generative AI performs when researching a topic outside the current news cycle. I tackled “Brexit” in my next search. After a couple of seconds, the system generated a sensible and, what I believe to be, accurate summary of the events.
One noteworthy aspect of generative AI is its ability to produce ready-to-use text. Curious, I probed further, asking for a 500-word rundown on Brexit. Instead of the anticipated summary, the search returned a list of links.
It’s almost as if Google wants to make it explicit that they don’t want to complete this task. The following example might illuminate the why.
Here is my next search term; “Can kids watch TV”? This time, I delved into one of the source articles. Instead of weaving an original narrative, the AI-generated summary mirrored sentences from the article almost verbatim.
If you made it this far to the article, chances are you have experimented even a little with ChatGPT. Since this was the first interaction many people had with generative AI, it has shaped the public’s expectations. As a result, the fact that Google doesn’t provide human-like written summaries fosters the impression that it lags behind its chief rival. It isn’t easy to overcome a user’s first impression about a product or feature, especially when it is not associated with a strong brand like Google. ChatGPT has raised the bar of what users expect — a reality we should all be mindful of when deciding what features to include in a product release.
This article aims to discuss best practices when building products that use generative AI. Here, in a nutshell, are the main learnings I hope you take away:
Transparency in Product Development: Transparency and managing user expectations are crucial in AI product development, especially when the system doesn’t behave as expected.
Cultivate User Trust: Building trust in a statistical model’s results is vital and becomes even more crucial when developing business applications.
ChatGPT vs. Your Product: When deciding which product features to implement, remember that ChatGPT has set the standards for what users expect from generative AI.
As we navigate the fast-evolving landscape of boundless opportunities, remember that these lessons can serve as the compass while you discover how generative AI can shape your product and business.
Thank you for reading! Let me know in the comments your thoughts and questions.
If you enjoyed this article, please give it a round of applause by hitting the clap button generously! Your support and engagement are greatly appreciated!
To stay updated on future AI insights and discussions in the business realm, follow the “AI with Lina” publication on Medium. For video content, follow me on YouTube. You can also connect with me directly on LinkedIn.
