User Experience Research with ChatGPT: My Life Hacks, Mistakes, and Conclusions
At Lamoda Tech, we’re all about user feedback — it’s the bread and butter of our decision-making. But let me tell you, getting to the A/B testing phase is a whole adventure. It all kicks off with getting into our users’ heads, figuring out what ticks them off, what gets them going, and chatting them up is key here.
As the lead of the research team at Lamoda Tech, my days are filled with interviews. But these aren’t just any interviews; they require a deep dive into empathy, an intricate dance with human psychology, and a quest to unravel the motivations that drive our users. And then, this idea hit me: What if we bring AI into the mix, like ChatGPT for analyzing, synthesizing interview results, and wrap-up?
I won’t sugarcoat it — it was a bit of a bumpy ride initially. But, at the end of the project, I nailed down a research plan and crunched through 24 interviews with the Jobs to be Done framework. The result? A whopping six days of work saved.
So, I took the leap, and here’s the scoop on what I learned.
Disclaimer
Don’t expect this piece to be your golden ticket to mastering research with ChatGPT4 — honestly, I doubt such a magic formula exists. What I’ve got here is a collection of my own adventures, peppered with both triumphs and face-palms. I’m laying it all out: the slip-ups, the lessons learned, the whole nine yards.
Let’s get one thing straight: GPT is no stand-in for a flesh-and-blood researcher. It’s a tool, plain and simple. And if research methodologies aren’t your jam, you’ll want to tread even more lightly with this tech. Sure, it can shave off some of your workload, but don’t think it’s going to do all the heavy lifting on its own. We’re not at the point where AI can tackle the trickier tasks without a human in the mix.
How ChatGPT4 Can Help
I set my sights on unraveling the mysteries of the “Beauty” section at Lamoda — think cosmetics, wellness goodies, and all things body care. Up till now, this corner of our world was more or less uncharted territory, with our knowledge of customer buying patterns pretty much a blank slate. Crafting a strategy for our “Beauty” lineup meant diving deep into the user experience.
Our plan? To peel back the layers of how users decide on and buy their fragrances, makeup, and skin care products through comprehensive interviews. We aimed to tackle a few key missions:
- Navigating the hurdles shoppers encounter while browsing our catalog, searching, and interacting with product pages.
- Dissecting the shopping journey from start to finish.
- Unraveling the mystery of what draws customers to a “Beauty” product page.
- Figuring out the must-have info shoppers want on these product pages.
- Create “jobs” based on the Jobs to be Done framework to get a clearer picture of our product’s audience
I planned to enlist ChatGPT4 for crafting respondent profiles, drafting interview scripts, and dissecting the feedback. But here’s a reality check right from the get-go: knowing your way around prompt engineering is great, but it’s not a silver bullet for every challenge
To get results from ChatGPT, communication is essential.
Kicking off this research meant getting the right mix of interviewees in the «chair». The big questions were buzzing: How many users to bring in? What slice should be from the Lamoda family? And what about sorting them by their budget, values, or views?
So, I handed over the reins of decision-making to ChatGPT, making that initial prompt according to all the rules of prompt engineering:
- I set the stage, giving ChatGPT a role to play — think of it as the research guru.
- I added context, spelling out that we’re diving into research on a hot topic, tying it to our business quests and what we’re aiming to uncover.
- The mission was laid out in verbs that meant business (think “act,” “determine,” “analyze”), zeroing in on what we wanted out of this whole shindig.
- Threw in an example for good measure, so the bot got the gist of what we were after.
- And didn’t forget to show how the result should look; leaving it to chance means you might get back anything in terms of the format.
I ticked all the boxes and followed every piece of advice to the letter. Yet, ChatGPT’s replies were on the shallow side before it hit a wall and checked out, overwhelmed by the info dump.
After a few swings and misses, I shifted gears. Opting for a divide-and-conquer tactic, I avoided bombarding it with everything all at once:
- I pared down the context to just the research questions, asking it to lock those in memory first.
- Next, I fed it the rest of the task details to chew on.
- Then, I nudged it to outline the criteria for picking our interviewees.
ChatGPT piped up this time, though the comeback was far from perfect. For example, it did not specify the exact number of people for the interviews, nor did it name the stores they should use.
But throwing in the towel isn’t my style. I kept at it, prodding the bot with pointed questions, determined to unearth the nitty-gritty details.
Eventually, I managed to cobble together a pretty decent set of criteria and came away with some key insights on collaborating with GPT:
1.Treat your exchanges with the bot like a dialogue, not a one-shot deal. Getting to the heart of what you need is more of a back-and-forth process, akin to a conversation with another human:
a) Prompt it for more details,
b) Seek out its take on things,
c) Challenge its responses when necessary.
2.If you’re dealing with a hefty load of info, introduce it to ChatGPT bit by bit.
3. And if the answers aren’t hitting the mark, don’t hesitate to probe further and express your concerns to the bot.
Check the Work — or Let ChatGPT Do It
Quality check is key, so I tasked ChatGPT next with assessing the criteria’s quality.
First off, I had the bot memorize the criteria list we’d drummed up. Then, I threw in the criteria set by Custom Research teammates used for a similar quantitative study, asking ChatGPT to weigh both and pick the winner.
ChatGPT tipped its hat to my colleagues’ criteria, ranking theirs a solid five against its own four. Curious, I pressed for reasons. The bot broke down the strengths of my colleagues’ criteria and where its own suggestions fell short.
Not one to settle, I nudged it for a polish-up on the criteria. Sure, it led to a few quirky suggestions and a bit of back-and-forth. Yet, when I laid out the revised criteria to the product team, the commerce folks, and the Custom Research crew, it was a thumbs up all around — no qualms.
Here’s what I walked away with:
1. Always have ChatGPT self-assess its output. You can do this by:
a) Pitting it against a higher-caliber example and asking for a comparison;
b) Casting it in the role of teacher or expert, soliciting a grade on its effort.
2. If it scores itself low, push for an upgrade. ChatGPT’s pretty good at stepping up its game.
Act in Stages
After nailing down the criteria for picking our interviewees, my next move was to draft up a research scenario. This isn’t just any document — it’s where we lay out our hypotheses, what we’re aiming to uncover, and the meaty business questions that drive our research. From there, we craft the actual questions we’ll spring on our respondents during interviews.
So, I fed ChatGPT everything: our research theme, the burning questions, our hypotheses, you name it, and asked it to sketch out a research scenario. The first stab was a letdown, honestly — ChatGPT tossed back questions that barely scratched the surface.
It hit me that I might’ve buried the bot under too much info. Time to break it down step by step:
- “Keep in mind, we’ve got a specific crowd in mind,” I reminded it, slipping our respondent criteria into the chat.
- “Here’s our laundry list of research queries. They’re numbered and ready to go. Let’s tackle each one, one at a time. What’s your question to unpack the first research query?”
ChatGPT and I took the questions one by one, and lo and behold, we started getting somewhere. I pushed for more depth, and sure enough, the quality of the scenario began to shape up.
I applied the same strategy to the hypotheses: served them up one at a time and quizzed how each would be tested.
Next, I wanted to merge the insights: meshing questions on the hypotheses with those on the research queries. But things got tangled — ChatGPT whipped up a muddle. So, it was back to break it down: I fed in the scenario as “scenario 1,” then did the same with “scenario 2,” aiming to blend them.
ChatGPT nearly got it right, barring a few hiccups — some questions vanished, others echoed themselves. After a bit of fine-tuning on my part, a solid scenario was ready to go. This exercise peeled back a few more layers on navigating ChatGPT:
1) When drowning in details, slicing the task into steps is smarter.
2) Load ChatGPT with as much as you can throw at it. Even stitching together separate answers into a unified whole is within its wheelhouse, as long as you’re clear with your instructions and vigilant about vetting the outcomes.
Translate from GPT Language to Human
After crafting our game plan, my team and I rolled up our sleeves for 24 deep-dive interviews. We leaned on the Riverside service service to turn our chats from voice to text, without me having to fuss over editing.
I tossed one of these transcripts ChatGPT’s way, curious about a few subtleties tucked within the lines. The bot didn’t miss a beat — catching my drift and shedding light on the nuances like a pro.
Then it was time to sift through the interviews with the Jobs to Be Done lens: pinpointing users’ tasks and desires regarding “Beauty” products. In this framework, these tasks and desires are dubbed “jobs,” with the idea being that users “hire” our product to tackle these jobs. A typical “job” might unfold as follows:
Template: As a [type of user], I want to [action] so that [outcome].
Example: As a driver, I want to use a reliable GPS navigator so that I don’t get lost in an unfamiliar city.
I introduced the bot to an article on the JTBD methodology, casting it as a researcher, and tasked it with unearthing the “jobs” lurking within our interview transcripts.
ChatGPT came back with a list of 7 or 8 solid “jobs.” Fired up, I rushed to share the breakthrough with my team, only to hit an unexpected snag: GPT’s dialect is, well, a tad robotic. While the “jobs” were technically on point, their descriptions were drenched in that unmistakable GPT lingo.
I’m a huge fan of “Surely You’re Joking, Mr. Feynman.” In it, Feynman, a renowned physicist and Nobel laureate, champions the idea that real mastery of a concept means you can break it down for a five-year-old. Inspired, I circled back to ChatGPT with a new challenge: “Pretend you’re chatting with a five-year-old. Can you simplify these jobs for them?”
Maybe my initial dive into the JTBD framework and the job examples wasn’t the best. I was on the verge of hunting down another example when it hit me: why not just ask the bot directly?
So, I kicked off a fresh conversation: “Can you break down the JTBD framework for me?” ChatGPT laid out the entire spectrum, from emotional to functional to social jobs and beyond. For my purposes, understanding functional and emotional jobs was key, and I requested examples. The bot’s take on it really clicked with me.
I fed the interview content back into the system, jogging its memory about who we are and our mission. I threw in one of its own example formulations for good measure and tasked it with pinpointing the jobs in the transcript. The improvement was night and day!
But there was still work to be done. I cast ChatGPT in the role of a university professor, asking it to critique each identified job and break down the reasoning behind its assessment. Plus, I wanted suggestions on how to enhance them.
Next up, I re-uploaded the roster of jobs alongside suggestions for refinement and called for tweaks. ChatGPT delivered, polishing off the transcription from our first interview beautifully.
Moving on, I kicked off a fresh chat for each new respondent, loading up their transcript and marching through the motions: pinning down the job, charting the hurdles in its way, mapping out the drive behind it, and, if it felt right, plucking out a quote to highlight.
Here’s what I’ve gathered from the experience:
- Interacting with GPT is akin to mentoring an intern: it’s keen to impress and loath to let you down. So, if it stumbles upon a gap in its knowledge, it might start to wing it. The key is to guide, not reprimand — ask why it took that path. It’s pretty good at taking a step back and owning up to a misstep.
- Laying down your ground rules is crucial. For example, If ChatGPT ventured into make-believe, I’d ask it to honestly say there’s no information next time it can’t find any. This approach actually bears fruit.
- Lastly, make sure whatever GPT conjures up isn’t just clear to you. Run it by your team or just someone else. It might not just be the wording at fault if things are still murky. Everything from how the task was framed, to the examples used, or even the job template itself could be throwing a spanner in the works.
Result
The endgame tally was a mix of 65 functional and 50 emotional jobs, with quite a few overlapping in essence. Guiding the bot, we distilled them into 4 core functional jobs and 4 key emotional ones.
Leveraging this streamlined approach, I dove back into the interview transcripts. My goal was to unearth insights that would answer our research queries and either back up or debunk our hypotheses regarding the product. I compiled these findings into my report and laid it all out for the team. And they were impressed! My report laid the groundwork for crafting the development strategy for the Beauty category at Lamoda.
What conclusions did I make?
- I’m eyeing the integration of ChatGPT4 into our research workflows, envisioning it as a powerhouse for sifting through in-depth interview data. As of October 2023, leveraging ChatGPT4 for UX research was still on the wish list. But, fast forward to now, and the tech has leaped forward, beckoning me to give it a whirl.
- Navigating ChatGPT4 effectively means ditching the expectation for instant answers in favor of engaging in thoughtful dialogue. It’s about posing questions and exploring possibilities together — “What if we approach it this way?”, “How about giving this a shot?”, “Are we on the right track here?”. This interactive process steers GPT towards delivering insightful outcomes.
- Incorporating ChatGPT notches up your professional game, streamlining research tasks remarkably. What used to be a week’s worth of analysis can now be wrapped up in a mere day and a half. Drafting up a design scenario? That’s down from a day to just 2–3 hours. It’s like having an exceptionally bright and capable assistant by your side. But why stop there? I’m branching out, and experimenting with Claude AI and Gemini via Poe.com, too.
- Diving into this tech opens up a playground of creative problem-solving. Assigning varied roles to the bot, I even had it channel the essence of a lead researcher from Usability Lab or step into the shoes of Don Norman, a luminary in user experience. Though, trying to emulate Norman didn’t quite pan out as hoped — maybe a language barrier?
- There’s something quite charming about how ChatGPT aims to assist, evoking the IKEA effect — you value what you’ve had a hand in creating more dearly. I admit, I got caught up in admiring GPT’s prowess without subjecting its output to rigorous critique. It’s a reminder of the importance of vetting work and seeking a second pair of eyes.
If you have a questions, comments or just want to talk about AI and Research, you can write me on:
Linkedin: https://www.linkedin.com/in/mike-yakovenko/
Mail: yakovenkom81@gmail.com
