AI vs. UX Researcher: Who Finds the Better Solution?
Introduction
As I delve deeper into generative AI, a question keeps nagging at me: how much of my future UX researcher role can be automated by AI? To explore this, I’m conducting an experiment to see if AI can match my capabilities in conducting exploratory research for product fit and solutions. This is both a quest to learn how I can leverage AI in my future work and a reality check to see if AI might make me obsolete. Overall, I want to see if AI can reach the same conclusions that I did through my research.
There’s a TLDR at the bottom for those that want to save time.
I’ll be comparing AI responses to research I conducted for a previous class project. Here’s a summary of the final product solution I reached.
Product Idea: Develop a comprehensive mobile experience for Ann Arbor residents and newcomers that streamlines purchasing fare, trip planning, and discovering new travel options.
Let’s move on to the experiment.
Experiment
The experiment itself is straightforward. I’ll use ChatGPT to conduct exploratory research on a hypothetical product. My goal is to see if it arrives at the same conclusions and solutions I did through my own research.
Experiment Tasks
- Conduct a Competitive Analysis to understand the existing landscape of similar products.
- Conduct Exploratory Research with User Interviews to gain insights into user needs and pain points.
- Provide Design Solutions Based on Research Findings to address user needs and differentiate your product from the competition.
What I hope to learn:
- Can I and ChatGPT reach the same conclusions?
- Will there be any discrepancies in our findings?
- What, if anything, did ChatGPT catch that I missed?
- Was there a clear advantage to me conducting the research myself?
- Can AI and humans collaborate to reach similar conclusions?
- Finally, the big question: will AI eventually replace UX researchers?
The Rules
I’ve set some ground rules for this experiment:
- I won’t edit any outputs from the AI.
- Prompts are my only tool for guiding the AI.
Tool
ChatGPT 3.5
Caveats
- Before we dive in, a few things to keep in mind. First, new research tools emerge constantly. I’m purposefully using ChatGPT 3.5 because it’s widely accessible and free (or very low cost) for most people. This focus on accessibility allows for easier replication and hopefully helps other researchers understand how AI can (and can’t) be leveraged in our work.
- This isn’t an attempt to definitively prove AI’s superiority over humans (or vice versa). It’s purely an exploration of its capabilities and the potential conclusions we can draw.
For brevity of the article I’m summarizing outputs. Original prompts and outputs can be found at the end of this section for reference.
Task 1: Conduct Competitive Analysis
Goal: Use the AI to conduct a competitor analysis similar to mine. This analysis included evaluating large, small, and local competitors to see if we reach similar conclusions.
Summary of Difference (Courtesy of ChatGPT)
ChatGPT Output: Focuses on the holistic approach of multi-modal apps, emphasizing user experience and understanding diverse user groups for app success.
My Output: Highlights the gap in current apps, missing alternative transit options like bike rentals and scooters. Emphasizes the need for comprehensive transit information to boost public transportation usage in Ann Arbor.
My Takeaway
The AI analysis didn’t include one large competitor, Google Maps. I’m not quite sure why but I did find that interesting! I also found that the pros and cons analysis felt quite high-level compared to my own, which focused more on in-app features. There were also some inaccuracies. For example, TheRide app isn’t operated by “TheRide” bus system, but by a third-party developer.
However, I did enjoy reading the AI’s perspective on “competitive advantages/standout features per app.” It piqued my interest in learning more about them myself. Overall, it was interesting to see the AI’s takeaways, even though they were much higher-level than mine.
Task 2: Conduct Exploratory Research with User Interviews
Goal: Based on what we’ve learned in the competitive analysis, come up with a potential design idea and validate user needs further with user interviews.
Summary of Difference (Courtesy of ChatGPT)
Both outputs address transportation needs for the ArborTransit Connect app but differ in detail and structure. Your version offers detailed frustrations and structured insights, emphasizing specific user needs and frustrations. ChatGPT’s output is concise and generalized, focusing on key insights without deep specifics or structured sections. Your approach provides a deeper understanding of user needs and frustrations, while ChatGPT’s is more high-level and straightforward.
My Takeaway
Again, I think ChatGPT is doing a wonderful job at providing high-level insight but is lacking depth and specificity when it comes to understanding the needs of Ann Arbor residents and visitors.
Task 3: Provide Design Solutions Based on Research Findings
Goal: Come up with design solutions to implement in my application based on learnings from competitive analysis and user interviews
Summary of Difference (Courtesy of ChatGPT)
ChatGPT Output: Focuses on general enhancements like real-time tracking, multi-modal integration, user-friendly design, and personalization.
My Output: Highlights specific features like in-app ticket purchasing, educating users on transit options, providing detailed transit info, and offering tailored experiences for residents and newcomers in Ann Arbor.
Differences: Your version provides detailed, structured features addressing specific user needs, while ChatGPT’s is more general and continuous in presentation, focusing on overall app improvements.
My Takeaway
There is a bit of cross over in our solutions, such as in offering personalization and integrating all available transit options. This is great to see!
Experiment Reflections
The future is bright with AI, and it’s growing faster than we can imagine. With ChatGPT 5 looming on the horizon, it’s difficult to predict how much better AI will become. Still, one thing is certain, as long as AI relies on historical data, there will likely always be a place for researchers who interview real people.
The biggest drawback I found with using AI was the high-level nature of its insights. It didn’t uncover any “golden nuggets” of information. There were also some inaccuracies, which is to be expected at this stage. By now, fact-checking AI responses should be second nature, especially if they’re going to be used to guide design solutions.
However, AI did excel at getting me up to speed on the competitive landscape and catching things I might have missed. For example, my research didn’t consider the “eco-friendly” user segment, and I completely overlooked Citymapper, a popular app I downloaded and loved after being introduced to it through the experiment. Additionally, ChatGPT churned out some excellent user interview questions that I could have incorporated into my own research.
While I might be biased, I believe my own research ultimately yielded stronger takeaways that could be directly leveraged into design decisions. I wouldn’t rely solely on AI analysis, but it did spark ideas about product features and user segments I hadn’t considered before.
This experience highlights the imperfections of both AI and humans. I overlook things(I’m only human), AI misrepresents some facts. At this stage of development, I see us as partners who can work together to achieve better outcomes.
Will AI take over my job?
Not yet…at least for now.
Thoughts?
So what do you think of this little experiment? Have you done exploratory research with AI successfully?
Note: I used AI to clean up the grammar, ideate article title names, and lastly create summaries throughout the article.
TLDR:
This summary was generated by ChatGPT.
The article explores the potential of AI in replacing the role of a UX researcher, using the author’s experience in a class project at the University of Michigan. The author tests ChatGPT’s capabilities against their own research methods in a three-part experiment.
Experiment Overview: The author uses ChatGPT for tasks like conducting a competitive analysis, exploratory research through user interviews, and providing design solutions based on findings.
Findings:
Competitive Analysis: While ChatGPT identified key competitors and their features, it missed Google Maps and provided high-level insights compared to the author’s detailed analysis.
User Interviews: ChatGPT’s insights emphasized real-time updates, multi-modal integration, user-friendly design, and personalization. In contrast, the author’s insights highlighted gaps in current apps and the need for comprehensive transit information in Ann Arbor.
Design Solutions: ChatGPT suggested general enhancements, whereas the author proposed specific features like in-app ticket purchasing, educating users on transit options, and tailored experiences for residents and newcomers.
Reflections: The author acknowledges AI’s potential in aiding research by catching oversights and providing a broader perspective on the competitive landscape. However, AI’s insights were found to be high-level, lacking in-depth analysis and sometimes containing inaccuracies. Despite its shortcomings, AI can serve as a valuable tool for researchers, complementing human efforts rather than replacing them entirely.
Conclusion: While AI has its strengths, including speed and data processing capabilities, it currently lacks the nuanced understanding and detailed insights that a human researcher can provide. The author believes that AI and humans can work together as partners to achieve better outcomes, but AI is not yet ready to replace the role of a UX researcher completely.
