2024 Summer of Learning -Integrating AI with User Experience Research
This summer, I worked as a User Experience (UX) Research intern with the Mainframe Software Division of Broadcom. As the first-ever Research intern on the Mainframe UX team, it was an exciting opportunity to learn from and contribute to the talented UX team behind the software for some of the most impactful systems in the world.
During my internship, I focused on a project titled “Smartly Integrate AI with UX Research,” which aimed to explore how artificial intelligence can enhance the UX process within Broadcom. Additionally, I worked closely with a Principal UX researcher on the Precision Design System, conducting a large-scale UX research project on the documentation site of the design system that is used across almost all Mainframe products. In this article, I will share my journey and findings from my 12-week internship experience with Broadcom.
Transitioning from Consumer to Enterprise UX Research
From a UX internship at a mobile app startup to a product internship at a B2B professional development services company, my previous work experiences focused on researching and designing products for individual consumers. As I transitioned from consumer-facing products to enterprise software, I found that the UX process required a deeper understanding of complex workflows, stakeholder needs, and organizational goals.
Enterprise software products on Mainframe computers are designed for the internal operations of companies, governments, universities, hospitals, industrial plants, transit terminals, etc. They are built to ensure smooth organizational operations by providing vital information, critical communications, and essential processes that help workers perform their jobs. The specific functions that need to be done using the Mainframe can be very specialized, presenting challenges for the UX team tasked with modernizing complex systems.
The stakes in Mainframe software UX design are high — decisions can impact millions or billions of dollars in potential waste or savings. Additionally, the scale of data that needs to be processed or monitored is unprecedented, with thousands or even hundreds of thousands of objects requiring numerous actions by users across multiple geographies.
Experiencing the Power of Team Collaboration in UX
At Broadcom, I also learned the power of collaboration among designers and researchers, rather than working as a solo contributor.
In many previous projects, I had to wear multiple hats — UX researcher, UX designer, project manager, and UI developer. While this approach works for small, independent projects, it often shortchanges the UX process and doesn’t leverage the unique strengths of a dedicated UX team.
During my internship, I observed researchers and designers from different value streams and products come together to solve problems collaboratively. Participating in large design critiques and reviews, where the entire team evaluates design solutions, was fascinating. I observed how researchers and designers from various products cycled through designs and gradually refined solutions to problems, and it motivated me to reach out to coworkers for feedback.
Investigating AI’s Role in UX Research
On the first day of my internship, I was assigned to a project called “Smartly Integrate AI with UX Research.” Although the current landscape is characterized by significant push towards artificial intelligence, I recognized that it would be premature to assume that AI was the most suitable solution. Instead, I chose to prioritize an in-depth analysis of the underlying problems and challenges, rather than rushing to apply AI as a potential fix.
I spent the first two weeks conducting informal stakeholder interviews with over a dozen people from the UX team to investigate the pain points in the UX process. Speaking with designers, researchers, and managers about their workflows not only provided insights into the UX process at Broadcom, but it also offered a valuable perspective on the career paths and approaches of others in the field.
Identified challenges included difficulties in finding participants and understanding user contexts, as well as communication gaps in the UX research process that were significantly hindering the workflow. It was this last point in particular, where I saw an opportunity for improvement using AI.
Researchers and designers often work on complex systems and analyze vast amounts of unstructured data, but managers and stakeholders can sometimes struggle to keep track of ongoing activities. Much meeting time is spent explaining context and updating others on the work being done, rather than optimizing time for feedback or exchanging new ideas.
I saw this as an opportunity to experiment with AI tools for summarization and rapid analysis to streamline the research process and enhance collaboration within the UX team.
Shifting UX Research from Reactive to Proactive with AI
Before discussing my implementation of AI as a summarization tool in the UX research process, it is important to understand why “AI in UX Research” matters. AI in UX research isn’t just about adopting the newest tools; it represents a shift from a reactive to a proactive design philosophy.
Traditionally, UX has focused on designing products that react to user cues. UX research investigates what users will do when they commit certain tasks, but with AI the goal is to design products that understand and execute the desired outcome. This paradigm shift of AI — from reactive to proactive software products — is pushing the field of UX design as well as UX research. While Mainframe software itself operates within a reactive paradigm based in methodical progression of tasks, its internal tools can benefit from this new proactive approach.
Research summaries should be generated before leadership requests them, creating seamless lines of communication. Analysis and code tagging should be dynamic, creating a foundation of insights for UX researchers to build off of. AI tooling should help UX researchers see one step ahead–that is the future of AI in UX.
Practical Applications and Challenges of AI in UX Research
When considering use cases for AI in UX research, AI’s strengths lie in coding, tagging to categorize similar findings, sorting through patterns, and leveraging extensive data sets. For my project, I focused on using these capabilities to create executive summaries from unstructured data gathered during user sessions.
Using LM Studio, a framework designed to deploy large language models on local machines, I experimented with various open-source models from Hugging Face by configuring prompts, temperatures, configuration files, and GPU settings. I tested the accuracy of the AI configurations by using raw, unstructured qualitative data from previous usability studies as inputs and comparing the summary output with what was produced by the researcher at the time.
In this iterative process of testing different models and configurations against a number of datasets, I created an “AI UX research assistant” that was able to produce executive summaries that had a high accuracy against executive summaries produced by researchers.
Case Study: Testing AI on Design System Documentation Research Study
To evaluate whether the AI improved the UX research process, I applied my findings from the experiments to a research project I conducted in collaboration with a Principal UX researcher at Broadcom.
The project focused on the documentation website for the Precision Design system, used by almost all Mainframe Software products. This involved creating a research plan and session guide, working with a designer on alternative designs, conducting interviews and usability testing, analyzing notes, and summarizing findings into an executive summary.
During this process, an obstacle was keeping the Principal UX researcher updated about each of the usability interviews she was unable to attend. With over ten user sessions and many tasks, it was challenging to balance my own note-taking with producing executive summaries for each session.
By feeding my notes and transcripts into LM Studio, I produced accurate and concise summaries in seconds — transforming communication between me and the stakeholders on this project.
Challenges and Future Directions
While I succeeded in generating AI summaries from my notes, I still encountered challenges. I found that while my notes were easily understood by the AI and produced reliable summaries and findings, summarizing from raw transcripts faced token limitations and often lacked the context needed to produce useful summaries.
Additionally, while the model handled notes in various document types and writing styles from different researchers, the phrase “garbage in, garbage out” proved true for working with AI. Notes that use a lot of shorthand or lack detail are not always “readable”, sometimes resulting in summaries that do not accurately reflect what happened during the session. AI cannot compensate for incomplete note-taking, which remains a human problem that technology cannot solve.
Ultimately, further research needs to be conducted to evaluate how AI tooling fits into the workflow of a larger team. Specifically, it is crucial that there are established channels of communication for this technology to be effective in improving workflows.
While challenges and limitations exist, I firmly believe that incorporating AI has the potential to greatly enhance UX research, particularly with respect to proactively generating summaries and readouts for leadership. In the future, I would be really interested to see how the same technology can be used for other areas of UX research, such as coding and thematic analysis.
Thank you to Broadcom for an incredible learning experience and for the opportunity to push the limits of UX research.
