Enhancing B2B User Research with AI: Introducing AnAÏs, Our Virtual Design Persona. (2/4)
Chapter 2: Testing the basic knowledge of generative AI tools
This article is part of a short series of 4 chapters exploring how AI is transforming user research at Akeneo (provider of Product Information Management ‘PIM’ software). The first chapter, introducing the topic, was published on October 26th, 2023 (Chapter 1: Context and Utilizing AI in User Research). In this initial chapter, we highlighted the primary challenges encountered in conducting user research within a B2B company. These challenges prompted us to explore new methods to achieve our research and design goals, particularly through innovative technologies like AI-driven solutions. This exploration led to the concept of creating an AI-based virtual persona named ‘AnAÏs’.
For this second chapter, our focus was on selecting the most suitable technology to embody a virtual, interactive persona. It provided an opportunity to evaluate the chosen AI’s foundational understanding of the Product Information Management market. The methodology for this initial experiment was as follows:
- Compare functional specifications from various free generative AI tools.
- Select at least two generative AI tools and compare their responses.
- Assess the basic knowledge of these generative AI tools regarding PIM.
- Choose the most effective AI candidate to serve as our virtual, interactive persona.
Which AI technology is the most relevant for this experimentation?
We had specific requirements for selecting our experimental technological foundation. We aimed for an AI text-based tool that was free to use as well as built on a robust large language model (LLM). To achieve this, we conducted a concise comparison (based on web articles) of the four most popular freely available LLMs to the general public: ChatGPT 3.5, Bard, Bing Chat, and Llama 2. Our comparison focused on functionality and practicality, prioritizing a conversational tool that closely mimicked real user interactions. Below is a summary table outlining our benchmark results:
We ultimately opted for ChatGPT 3.5 from OpenAI and Google Bard to proceed with the initial phase of our experimentation. This decision was clear-cut as both platforms offer user-friendly interfaces and deliver sufficiently accurate and complete responses during extended conversations. This capability is vital for simulating genuine interactions, mirroring the experience of conversing with a real user. While they do have limitations, these are far less restrictive for our requirements compared to the constraints observed with the other two LLMs.
Generative AI tools already know a lot about Product Information Management (PIM)
After we had chosen our two technological supports for our experiment, we saw what they were capable of when it came to questioning them about product information management, Akeneo, and its ecosystem.
Since we hadn’t yet trained these AI tools to act as our virtual persona, for the purpose of this chapter, we’ll refer to them as ‘naive’ AI. Initiating from a clean slate in our conversations, the themes and questions presented to ChatGPT and Bard served a dual purpose:
- Assess their existing knowledge
- Provide them with the specific context required to become an accurate virtual persona
This phase marked the initial step of their training journey.
Given the rather long responses from ChatGPT and Google Bard to our preliminary questions, we have decided to only present our comparative analysis without exposing the full answers for the sake of readers.
Assessing General Knowledge About PIM: Our Initial Step
We initiated the discussion by presenting two broad questions to ChatGPT and Bard, delving into the realm of product information management and its broader ecosystem. This approach served as an introduction to the topic at hand.
1. What do you know about the PIM (Product Information Management)?
2. Can you provide information about the PIM ecosystem?
In both instances, Bard’s responses were more extensive and comprehensive, offering a broader perspective on PIM and its ecosystem. While ChatGPT focused primarily on detailing 8 key features of PIM software, Bard expanded its response, discussing the software’s benefits, core use cases, and highlighting 5 essential features (data modeling, import & export, enrichment, data quality, and distribution). However, when examining ChatGPT’s responses at a granular level, they exhibited a more thorough and precise understanding of each aspect covered.
Regarding the ecosystem question, ChatGPT outlined 12 key components ranging from PIM software providers to various integrations (e-commerce, marketplace platforms, DAM, CMS, ERP, CRM, workflow tools, etc.) and related services like consulting and training. Conversely, Bard provided additional examples of the same components and expanded on the PIM’s role within the omnichannel ecosystem, trends in the PIM market, and offered an optimistic view of its future.
In these general questions, Bard’s responses proved more pertinent as they offered a comprehensive view of PIM without unnecessary specifics. This broader perspective favored Bard’s contributions.
Evaluating Their Understanding of Akeneo: The Next Inquiry
We then delved into the subject with more targeted and precise questions, beginning with the products and features offered by Akeneo. Our aim was to gauge the depth of knowledge possessed by these AIs regarding our company, our current product lineup, distinctive features, and insights into the market. We posed the following three questions:
3. What do you know about Akeneo PIM?
4. In your opinion, what features are missing from Akeneo PIM?
5. What are Akeneo’s advantages over the competition?
Both ChatGPT and Bard presented similar responses to the previous section on introducing Akeneo PIM, albeit with slight variations. ChatGPT highlighted distinctions like the open-source and enterprise licenses, while Bard included the Akeneo website URL. However, when addressing missing features, ChatGPT excelled by addressing almost all the feature gaps identified in our two years of user research, aligning with several elements already in our product roadmap. On the contrary, Google Bard suggested improvements to existing features, missing the mark in addressing our specific question and failing to provide valuable insights.
In comparing Akeneo to competitors, ChatGPT impressively outlined 9 detailed advantages, encompassing user-friendly interfaces, an extensive connector marketplace, scalability, and the Product Experience Management (PXM) approach. In contrast, Bard’s response was less comprehensive, offering only 6 advantages, lacking the depth provided by ChatGPT’s insights, which we had already received.
Ok but… What do they know about PIM users?
In our quest to simulate real user interactions, our focus shifted towards understanding their comprehension of PIM software users — examining their profiles, roles, needs, and challenges. Isn’t this, after all, the primary objective of a persona tool for all designers and researchers?
6. Who are the main PIM software users?
7. What is a typical day for a team managing product information?
8. What are the challenges for a Product Manager working with a PIM tool?
ChatGPT provided an extensive list of 10 precise user profiles within an organization that could use a PIM software. Conversely, Bard listed only 6 generic profiles or teams involved in PIM software usage, lacking the depth and specificity of ChatGPT’s insights.
Regarding the typical day of a team managing product information, ChatGPT adeptly recognized the variability influenced by organizational size, industry, and internal processes. It offered a generalized overview of the main jobs to be done rather than a detailed hour-by-hour breakdown, a contrast to Bard’s approach, which included irrelevant specifics like lunch breaks or leaving the office.
In outlining challenges faced by these teams, ChatGPT presented 12 concrete examples that were pragmatic and grounded, contrasting with Bard’s 10 more abstract challenges (such as staying updated with technology trends, managing stakeholder expectations, and balancing agility and stability). For instance, while Bard discussed managing change and adoption, ChatGPT delved into specifics like user adoption and training.
Once again, ChatGPT’s responses were more pertinent and aligned with our expectations.
Let’s talk about real topics… What about “workflows”?
Ultimately, we needed to select a specific topic previously explored in user research to compare ChatGPT and Bard’s responses against those of real users. We opted to inquire about the PIM Team workflow — a recent, intricate subject rich with challenges and potential. Importantly, this topic wasn’t directly tied to our product usage, eliminating the necessity for familiarity with Akeneo PIM interfaces to respond.
9. Who are the different teams involved in the content enrichment process?
10. What are the different tasks of the content enrichment process? Who is responsible of completing each task?
11. Could you provide more information about content review or validation tasks?
12. How Product Managers know when they need to complete a task?
ChatGPT meticulously outlined the various teams involved in the content enrichment process, detailing 10 teams and describing between 1 to 3 roles per team along with the mission of each role, totalling 21 roles. In contrast, Bard introduced 2 teams and 7 roles, albeit in a way that made it challenging to comprehend, despite presenting the main missions of each role in a summarised table.
Regarding the steps of the content enrichment process, ChatGPT provided a clear and comprehensive workflow, breaking down 29 tasks across 14 steps, offering specificity and relevance. Conversely, Bard’s workflow was more generic and high-level, comprising 19 tasks within 5 steps, lacking the granularity and depth present in ChatGPT’s breakdown. Similar observations arose when examining the content review and validation process.
Finally, both platforms failed to grasp our last question adequately. We sought insights into notification needs, but instead, received information regarding the steps and tasks in which the Product Manager intervenes. This departure from our expected response was consistent across both ChatGPT and Bard’s outputs.
What did we learn from this preliminary study about the basic knowledge of these AIs?
And the winner is… ChatGPT 3.5! As anticipated from the comparative table showcased in the initial segment of this article, ChatGPT delivered answers that were more precise, comprehensive, and practical, contrasting with Google Bard’s tendency to remain at a broader, more generic level (admirable for the initial broad questions). This distinction solidifies our belief that ChatGPT 3.5 stands as the prime choice to evolve into our first AI-based virtual persona, AnAÏs.
Moreover, the quality of responses from both ChatGPT and Bard pleasantly surprised us, offering rich insights into PIM and its ecosystem, along with valuable user perspectives, despite operating as ‘Naive AI’ without specific training in our experimental context. These impressive performances have further reassured us in the idea of continuing on this path, in order to best exploit these technologies to overcome the challenges of B2B user research. Given these remarkable outcomes from naive AI, we can’t wait to see what it will be like after a little training.
The roadmap ahead is clear — through an iterative process, we aim to enrich ChatGPT 3.5 with genuine insights gleaned from Workflow user research. We’ll compare its responses against real user input, gradually providing more context. Note that with a paid licence and the new feature called ‘GPTs’, all information remains private and secure.
Stay tuned for our forthcoming article in this series, as we delve deeper into enhancing B2B User Research with AI!
*An article written with the help of generative AI (ChatGPT 3.5).
