Experimenting with GenAI in Design Research
The use of generative AI in design processes has been a contested topic since its adoption into mainstream technology. Should we use it? If yes, to what extent? Does it make our work as designers more or less complicated? Is it even ethical?
Amidst the unexpected removal of OpenAI’s (now) ex-CEO Sam Altman over a week ago, opinions on generative AI’s foundations and future are growing more divided. Those with an unwavering belief in its potential have advocated for its many uses: accessing vast and varying data pools for research, making data analysis quicker, pushing the limits of imagination when ideating solutions, etc. The list is long and the possibilities are plenty (here is a great article by Samuel Tschepe outlining some use-cases in design thinking).
However, much of the conversation among designers is speculative. As with any tool, the quickest trick for new users to find personal success is to conduct actual experiments. And this is exactly what Cameron Hanson, Strategy Director at Smart Design, and her Brooklyn-based team did. She explored three particular stages of the design research process, and experimented with how GenAI could be a part of each.
The results she presented at the recent talk organized by Service Design Network (New York Chapter) got us all chatting about whether we too could be incorporating AI in our own work, and what we should be wary of.
Experiment 1: Can AI mimic a user?
The first gap in the design research process that Cameron identified was the beginning stage, when you are entering your client’s world, and need help understanding what their user’s needs are. Can AI successfully convey what a potential user would? Or in other words…
How might synthetic users strengthen the understanding of our participants?
For this experiment, the Smart Design team used a research study they had already been working on, one that asked participants how they used technology for health and wellness. Bard, Google’s AI, was asked to mimic a participant who was on top of their health and led a holistic lifestyle.
The interview results were accurate– Bard had successfully answered the questions in a manner that matched the human users’ responses, and the responses were easy to scan. However, the more Cameron’s team engaged with Bard, the more “artificial” the intelligence got. Follow-up questions were met with generic, non-emotive answers. The more you ask, the more general content it adds, which sounds like an accumulation of several users’ experiences and therefore starts to read as unrealistic.
Switching up the user profile made it worse. When asked to mimic a user who struggles with their health and needs help, Bard got confused. It claimed to have an extensive mindfulness routine, including a meditation app, mood tracking app and online support group. The rate of miscommunication seemed to be high enough to negatively impact the research process.
A notable observation is that GenAI is unable to provide any anecdotes during interviews; there are “no nuggets that summarize what the tension is”. To an extent, GenAI can provide the “what”s but not the “why”s of a user’s experience.
So, the verdict? Use GenAI to get up to speed faster, and for testing research questions and interview flows. But do not use it to substitute research with actual humans.
Experiment 2: Can AI speed up research synthesis?
After the research is done and design researchers are swimming in data, is there an efficient way to parse through it all and prioritize information?
How might document queries propel us to deeper insights?
A query is a request to AI, and its purpose is to refer to data, perform calculations and provide answers. To feed the specific data into the AI, Cameron’s team had to scrub all their documents and replace all recognizable information from their research material.
The query that was asked was– “What is the perspective of US healthcare currently?” The team found that specific queries got them closer to insights, but over-indexed on keywords. This means that the colloquial nature of interview responses does not tend to be accounted for and the essence or purpose of the response that would lead to an insight could easily be skipped if too many buzzwords are used. AI also completely missed nuance, sarcasm, and humor.
With AI, biases come into play which are difficult to mitigate. As an audience member said after the talk, we as human beings are biased. The information we have is privy to bias, so how can any machine learning tool created by and dependent on human perception be free of bias?
On the flip side, the access to varied databases gives processing with GenAI the opportunity to challenge our own biases as design researchers. While Cameron cited this as a possibility, she herself was not challenged by any surprising information through the course of the experiment.
This experiment yielded a divided conclusion– there’s a risk to privacy and security, considerable time investment and lack of insight accuracy. Most importantly, GenAI understands words, but not meaning.
Experiment 3: Should AI have a seat at the ideation table?
The popularity of image generation and programs like Midjourney has skyrocketed in the past year, with its ability to produce whimsical, illustrative art pieces as well as high-quality images for marketing campaigns. But can we reverse the process and use it as a jumping-off point instead of a finished product?
How might text-to-image enrich initial ideation?
For this final experiment, Smart Design strategists who were already well-versed with Midjourney, and had experience writing effective prompts, were gathered at the ideation table. They sat with a hypothetical question that asked how we could attempt to have a rat-free NYC.
With a challenge like this, there is space for some crazy ideas, but the immediate problem that emerged with text-to-image ideation was that the process was no longer scrappy. The rawness of scribbling on post-it notes and drawing stick figure sketches was gone.
Instead, too much precious in-person time and energy was being wasted on perfecting the language of the prompts, and being distracted by the very well-made but far-off visuals. It is frustrating, especially when you don’t yet know the specifics of the idea you are trying to visualize. Some of the images were downright absurd because a vague prompt will yield unhelpful imagery.
GenAI is hypothetically good for provocations, actually helpful in imagining future worlds, and best at perfecting final-stage mockups. It is not a collaborative tool and isn’t suitable for real-time sessions.
However, Cameron found that co-ideation might be a better strategy; once a few ideas are settled on, pairing strategists who doesn’t know Midjourney well with someone who does could be worth exploring.
To recap:
- GenAI has a lot of answers but can’t enable empathy.
- GenAI understands words but it doesn’t understand meaning.
- Text-to-image generation is better suited behind the scenes, it’s not ready for real-time ideation.
GenAI is not human and it does not have a lived experience.
Brands that are jumping on the AI train are definitively fast-tracking their internal processes and churning out material, but it is important that as designers, especially in the research phase, we do not sacrifice the value of keeping human involvement at the forefront.
