I used ChatGPT in one of my user research projects and here is what happened
In the realm of User Experience (UX) research, where human insights steer the course of design, a new ally has emerged from the digital landscape: artificial intelligence, in the form of ChatGPT. It’s only natural to wonder as a researcher how an AI chatbot can enhance the very processes of user research and drive its evolution. In this article, I will explain the journey to investigate the potential of integrating ChatGPT into every step of the user research process, uncovering its promises and pitfalls alike.
The allure of AI, with its rapid cognitive abilities and extensive data processing power, called me to explore its application in realms like user research, previously reserved for human translation and intuition. The user research process, a cornerstone of user-centered design, presents a compelling arena for such exploration. By harnessing the capabilities of ChatGPT, I wanted to magnify our understanding of user behavior, refine our product development strategies, and ultimately sculpt digital experiences that seamlessly align with user needs and desires.
Imagine the possibilities: conducting in-depth user interviews facilitated by AI, analyzing and extracting trends from vast amounts of textual data in seconds, and even ideating and prototyping with the assistance of a tireless digital collaborator. Sounds like a dream, right? ChatGPT’s ability to generate human-like text, comprehend complex queries, and provide contextually relevant responses opens doors to novel research methodologies that could redefine the way we understand users.
However, I must note that while I was enthusiastic about the potential capabilities AI offers when I first started on this journey of testing ChatGPT for user research, I still had the belief that AI alone could not possibly fully undertake a user research project from start to finish. User research necessitates the human touch — after all, we, as user researchers, are conveying human stories to products, and an AI tool could not possibly capture the nuances and complexities of human narratives. Or so I initially thought.
Let’s dive into my process:
Exploration
In crafting the foundation for my exploration, I carefully selected a study that employed a comprehensive qualitative survey approach with a qualitative inclination. Therefore, the study I chose for my journey of integrating ChatGPT into user research wasn’t arbitrary; it was a deliberate decision that aligns with the essence of user research. My goal was to assess the potential of ChatGPT in a scenario involving extensive qualitative surveys. In other words, my choice of this study lay in the extensive volume of collected data due to the methodology selected. The study, characterized by its extensive data collection, involved parsing through the complicated details of responses provided by hundreds of users. I wanted to determine if ChatGPT could assist in deciphering patterns, identifying correlations, and extracting overarching themes from the data, similar to what I could achieve with affinity diagramming, analysis and other techniques.
While experimenting with prompts for ChatGPT, I quickly realized that, regardless of the purpose for which I used ChatGPT, the initial iterations of my prompts never produced the desired results. To consistently achieve accurate outcomes, I found it necessary to refine not only the content of my prompts but also my communication style with the AI and the guidelines I provided. This iterative process proved crucial in harnessing the full potential of ChatGPT.
I quickly realized the importance of providing explicit guidance to the chatbot. It became evident that I needed to thoroughly explain the chatbot’s role in the task, define its objectives in as much detail as possible, clarify the methodology it should employ, and establish precise guidelines for the desired outcomes. In essence, the key to unlocking ChatGPT’s full potential lay in micromanaging its interactions to an extraordinary degree.
Writing the research plan
One of the most pivotal points in any user research project is the creation of a well-structured and purposeful research plan. Therefore, after defining the problem space to ChatGPT, I invited the AI to generate research questions and goals that would steer the study towards actionable insights, as well as come up with methodology planning –even though based on my expertise I knew most appropriate methodology according to our timeline and scope would be the qualitative surveys, I wanted to see if ChatGPT could come up with alternatives.
To do so, I prompted the chatbot with
“Imagine that you are an expert user researcher, currently working on a project with a focus on [problem space]. Also, imagine that youwant to write an extensive research plan, specifically highlighting the research goals, research questions, potential impact of the study, and the research methodologies.
Here are the scope and constraints of the project:
Problem Space: [Provide a concise description of the problem space]
Target Audience: [Specify the target audience]
Context: [Provide any relevant background information or context]
Now, please write the extensive research plan for the [problem space] project as a high-level user researcher.”
However, the initial responses I received from ChatGPT were indeed insightful, yet somewhat generic in nature. Recognizing this, I realized that for ChatGPT to generate more project-specific and targeted suggestions, I needed to prompt it with a more refined context, by including the hypotheses I want to test, information/data that led us to our focused problem space and chatbot’s role that I am aiming to target. This is when I started re-iterating my prompt for my goals, which took about 6–7 more tries by adding additional context and changing the way I included that context.
To achieve this, I rephrased the explanation of our problem space multiple times (using different words and phrasing) and explained my role, the level of expertise, the exact outcome’s outline I want, and what I want to achieve, with its specifics. I also incorporated more specific scope and context in my prompt. This nuanced adjustment worked better, as ChatGPT’s responses showed some improvement in relevance and precision.
Shaping the narrative to provide ChatGPT with a more focused context through different prompt trials, helped me effectively channel its powers toward our project’s specific needs. This realization underscored the importance of iterative refinement — of not just the AI’s responses, but also the inputs we feed into it. ChatGPT’s understanding evolved alongside the project’s requirements through this iterative process, ultimately yielding insights that aligned harmoniously with the study’s objectives and scope.
As we delved deeper into the project, I discovered that the ChatGPT could also be a guiding hand in formulating strategic survey questions. By incorporating more contextual details and prompting ChatGPT with a specific task such as:
“Imagine you are a highly experienced user researcher. You are planning to conduct a survey to explore user perceptions and preferences regarding [problem space,]. Your primary research goals are [ ].
The survey should adhere to the following guidelines:
[List any specific guidelines, constraints, or scope limitations]
You aim to structure the survey questions as follows:
[Explain the question structure, e.g., ‘mix of multiple-choice and open-ended questions, with a focus on usability and satisfaction’].
Please provide a set of survey questions that you would use in this user research study.”
I found the AI’s responses to be remarkably insightful and action-oriented. This tailored approach facilitated the generation of survey questions that were not only attuned to our study’s objectives but also aligned with the diverse facets of our user research goals.
Analyzing the data
ChatGPT definitely made analyzing qualitative mass data more manageable through the strategic use of AI assistance. The dataset was a mix of diverse inputs, ranging from comprehensive insights to unqualified responses and outlier entries that added to the complexity. To harness ChatGPT’s analytical prowess effectively, a preliminary phase of data cleaning was essential. Outliers and out-of-place inputs had to be manually sifted out before presenting the dataset to the AI. This preparatory step was crucial, as it paved the way for ChatGPT to comprehend and derive meaningful insights from the nuanced qualitative data. Otherwise, ChatGPT failed at evaluating the data.
In addition to refining the data, effective communication with ChatGPT proved to be a critical factor. The AI’s ability to process information hinged on clear instructions — letting it know that each row represented different user inputs, and even providing contextual explanations about the responses being analyzed. This interactive dialogue ensured that ChatGPT had the necessary information to navigate the intricate landscape of qualitative data analysis. For that, I prepared prompts for each survey question I asked, such as,
“You are a very highly experienced user researcher. You recently conducted a qualitative survey on [problem space ]. In the survey, you wanted to learn [research goals].
From the dataset provided below, where each row represents different users’ survey responses, your task is to analyze the user data specifically related to the question: [add the survey question here].
The analysis method you intend to employ is [e.g. affinity diagramming], with the goal of [e.g. identifying trends and patterns within the responses]. Your ultimate objective is to uncover the top patterns and outcomes from the data, supported by user quotes as illustrative examples.
[Insert the dataset here, ensuring it accurately represents the survey responses.]
Please proceed to identify and describe the most prominent patterns and outcomes you discern in the data, using user quotes to exemplify your findings.”
While ChatGPT exhibited strengths in identifying consensus and saturations within the qualitative data, there were some limitations. Calculating the percentages of responses was a challenge for the chatbot, which required a more structured and formulaic understanding. However, its capability to unravel patterns, pinpoint common themes, and extract the essence of saturations from the data was pretty beneficial. The AI’s insights acted as a guiding compass, directing attention towards prevalent themes and allowing for a more targeted approach to in-depth analysis.
Turning Insights into Actions
I wanted to explore the full potential of ChatGPT in the user research process, so I was excited to test its capabilities in a more advanced arena: translating insights into actionable business items. With that, I wanted to extract tangible steps from the found insights. However, as the experiment unfolded, it became evident that even ChatGPT had its limitations.
Here, ChatGPT’s performance was different from its successes in earlier stages of the user research process. While it had greatly assisted in crafting research plans, generating survey questions, and even uncovering saturations in qualitative data, it stumbled when it came to recommending actionable business items. The outputs it provided were disappointingly generic, offering suggestions such as “Usability Enhancements”, “Engagement Enhancement,” “Related Content Suggestions,” or “Focusing on User-Generated Content.” lacking context that would have made them truly actionable and aligning it with our product development process.
It became clear that ChatGPT’s shortcomings were tied to the depth of context I provided. The process of translating insights into actionable business items demanded a nuanced understanding of our business strategy, market positioning, and prioritization imperatives. While ChatGPT excelled in data analysis and pattern recognition, the step of distilling these insights into coherent and contextually fitting actions required a level of strategic thinking that, at least for now, remains rooted only in human understanding.
Overall Opinions
In reflecting on this journey, it’s important to acknowledge that my exploration of ChatGPT’s capabilities was conducted within a specific context. I delved into the realms of user research with a single evaluative study (qualitative survey) as my canvas, and my opinions shared here are bound to that particular canvas.
This journey showed an exciting opportunities and options, revealing how ChatGPT can be a helpful partner in different parts of user research.If one can train themselves through experience on how to prompt the chatbot effectively, exercise patience through constant iteration, and provide the necessary specifics and details, ChatGPT can be a valuable tool for automating manual tasks in user research, too. ChatGPT can be good at tasks that need to be done quickly, dealing with lots of information, and even coming up with new ideas. These achievements were really impressive and highlighted how AI can work together with us, making things more efficient and letting us look at things in more detail.
But as this journey went on, it became clear that there are limits to what AI can do right now. Even though ChatGPT showed how good it can be in some areas, it struggled when it came to tasks that needed carreful strategic thinking and putting different things together. To put it simply, there are parts of user research that are like tricky puzzles, and these need the kind of knowledge that humans have from their experiences and understanding different situations.
In conclusion, my experience using ChatGPT in this study shows both the exciting possibilities and the limits of AI in user research. This adventure highlighted how humans and AI can work together to understand things better. As I look back on what I’ve done, I’m both excited about what’s coming next and aware of the journey ahead. This journey is where AI keeps improving, but human understanding is still really important.
Overall, I enjoyed the way AI lends its hand to streamline mundane and time-consuming tasks, leaving me to take on strategic challenges. I enjoyed playing around with AI and AI prompts for this project so much that, who knows, I might have even asked ChatGPT to lend a hand in writing this article!
Disclaimer: ChatGPT’s GPT-3.5 version was used in this project.
