Personalization (Part 1 of 3): Generative AI Persona Types
Our interactions with Generative AI chatbots are largely useful and entertaining. There are noteworthy exceptions. Failure modes for Generative AI include when their outputs are inaccurate, unconvincing, or both. If the output content does not appear to address a prompt’s point of interest completely or accurately, the user will be frustrated by the lack of useful information. Similarly, if the framing and composition of an output isn’t tuned well — for instance the content is accurate but the Generative AI expresses uncertainty about its veracity, the user may doubt the capability of the AI. Even highly advanced chatbots such as ChatGPT Plus (based on OpenAI’s latest and greatest LLM, GPT-4) are still prone to error. Chatbots based on GPT-4, such as Bing Chat, have additionally demonstrated unusual behavior that some users found quite disturbing.
In a prior piece, I explored the issue of why Bing Chat may have demonstrated these kinds of failure modes in its first two weeks of public release (before user engagement with the chatbot was significantly constrained by Microsoft), despite being based on a state-of-the-art large language model (LLM). Beyond known issues in LLM-based applications regarding inaccuracy and hallucination, Bing Chat exhibited intemperate personality traits, in various instances expressing condescension, self-pitying and resentment. I postulated that incorporating both dynamic Web search results and elements of a previously trained Microsoft chatbot (Sydney) were likely behind the reported negative user experiences. Given the issues we’ve already seen, how can we evaluate more systematically the risk of psychological harm to users that may be caused by a chatbot? This is a central question I tackle in a longer, recently published pre-print article. The article explores the relational dynamics between LLM-based chatbots and human users in order to better contextualize why different AI persona types have distinct design considerations and may pose different levels of psychological safety risk. This piece (the first part of three) sets the stage for that longer analysis.
Because Generative AI chatbot interactions are conversational in nature, personalization dynamics must also take into account an intent by a human user that an AI’s outputs reflect the appearance or actuality that the AI is getting to know more about that user. In turn, convincing outputs from an AI that are or appear personalized to a user may cultivate a feeling by the user that they are getting to know the AI, its unique traits and persona. Sustained interactions of this kind are how humans get to know and build affinity with each other. The sense of getting to know and being known and understood has considerable psychological effects for most humans. It is how we get comfortable with people with whom we are engaging, whether in a fleeting, transactionally-driven encounter (such as interactions with a restaurant waiter) or in a more durable, long term relationship (such as with a spouse or business partner). Although most people may be able to maintain cognitive separation — the knowledge that a chatbot is just a machine, even while getting personalized benefits from an interaction with it — many may be prone to attributing human personas to a chatbot, becoming relationally attached to it. (I devote a section in the pre-print article to explaining more about the known psychological dynamics of chatbot/human interactions.)
For present purposes, I use the term “personalization” to mean:
- an intent by an AI (or its operator) to give the appearance of knowing, or to actually undertake to know more about, a particular user, or
- the effect of an AI’s outputs (even if spontaneous or inadvertent) in conveying such personal familiarity
As noted, LLM-based chatbot systems are designed to be dialectical. They are trained to make sense of language presented to them by a human in the form of a prompt, then generate responsive language to satisfy the objective of the prompt. There may be several prompt/output cycles in a given session with a chatbot, with each cycle building on the context set by prior ones. Each cycle, then, must maintain some topical continuity with those preceding it in order to preserve a sense of conversational coherency. For that reason, LLM-based chatbot applications must be designed to exhibit conversational proficiency. Part of that proficiency is how engaging the Chatbot is perceived to be by its users. Engagingness, in turn, is influenced by personalization goals and how well the personalization design of a chatbot meets user expectations. While all LLM-based chatbots are designed to be personalized, the mechanics of how this is accomplished and specific personalization goals differ according to the type of AI persona a chatbot is designed to present.
LLM-based chatbots currently being deployed for public use have been designed to fulfill a number of persona objectives. These range generally between Generative AI chatbot as a robotic assistant on one end of the spectrum and Generative AI chatbot as a human surrogate on the other.
- AI as Robotic Assistant: The overall goal is to have the chatbot meet the informational or assistive desires of a user. The user wishes to obtain an advanced form of search service, one that can not only obtain information and knowledge but that can also synthesize them in new and interesting ways and output the results in a variety of formats meeting user specifications. These goals are paramount and no relational dynamic is sought or desired. The AI is seen for what it is: a computerized retrieval system with the capability to produce highly customized, sophisticated outputs.
- AI as Human Surrogate: The overall goal here is to have a chatbot meet the human relational desires of a user. The user presumably wishes to form a deeper bond and emotional or relational connection with the chatbot over time. The primary design goal for such chatbots is to have them present human personality traits in every user interaction. The design goals may further include having the chatbot develop a detailed, more nuanced understanding of a user’s personality over time and, in turn, present specific, customized personality traits in response to those exhibited by the user. Emotive or feelingful interactions may be part of such design goals.
Of course, these persona objectives are not discrete. LLM-based chatbots may be designed to be both relationally human-like and assistive. LLM-based chatbot personas spanning the spectrum of persona objectives (as well as examples of each variety currently in deployment) include:
- AI as Assistant/Instructor: We have many people in our lives who we keep at arm’s length relationally, though we appreciate them because they provide critical help to us. These include services providers (such as accountants, doctors, lawyers, concierges, research assistants), instructors (tutors, teachers, professors), and guides (editors, coaches). We look to such persons as sources of information, expertise, contribution, or specialized knowledge that help us problem solve or get important things done. The more prominent LLM-based chatbots such as ChatGPT, Bard, Bing Chat, and Claude 2 provide this form of arm’s length expertise and support to their users.
- AI as Butler/Mentor: We can sometimes form friendships with people with whom we are in a services relationship, or to whom we look for guidance and wisdom. A butler may primarily provide personal assistance and services, but may also become a close family friend. Similarly, a mentor is someone to whom we turn for sage advice and insight but with whom a close or trusted relationship may develop. Services such as Inflection’s Pi.ai provide a “personal ai” to each of its users, with the expectation that users may “share some of their deepest held ideas, desires and personal information.” Pi.ai seeks to provide AI companions “with the single mission of making you happier, healthier and more productive.”
- AI as Friend: Friendships can be casual or also very close and intimate. As such the category of “AI friendship” itself has a broad range of possibilities. Services such as Chai AI aim to address users who may be lonely, and to provide them with entertaining exchanges and even friendship over time. Some users may seek romantic partnerships from such chatbots, while others may seek an occasional novelty interaction with them rather than a sustained relationship with high frequency exchanges. Character.ai is an example of the latter, whereby a user can create a chatbot persona (replete with an on-screen avatar) either ad hoc or to resemble a celebrity or publicly known figure, largely for entertainment purposes. Some users, however, may become more attached to the personas they create than others. It remains to be seen how Meta’s forthcoming AI personas will be designed and deployed on Facebook or other properties.
- AI as Romantic Partner: Romantic relationships are in many respects the most intimate, vulnerable, and close kinds of relationships that humans develop. They involve a range of emotions, the formation of close relationships, and psychological attachments. This is overtly the objective of services such as Replika.ai.
- AI as Personal Surrogate: At least once, you may have heard someone joke that they wished they had a clone of themselves. That person was probably over-extended or simply wished they could do more things simultaneously. Services such as Personal.ai and Otter.ai seek to help users create digital extensions of themselves. Such chatbots are designed to respond to prompts using one’s own facts, opinion and style (Personal.ai) and to stand in for their human counterparts at meetings (Otter.ai).
Why do such distinctions between AI personas matter and how do they allow us to think more systematically about matters of chatbot proficiency, design, and safety? Imagine that a chatbot connected to a software agent that can complete food delivery orders has previously been informed that I’m allergic to shellfish. If I ask it to order dinner and it orders a shrimp meal for me, we would deem this a failure of personalization execution. Putting aside whether my shellfish allergy is acute or not, the dinner order wouldn’t necessarily create a safety issue of the kind we’re focused on.
Consider, however, another scenario. A chatbot you just prompted to identify the year humans first landed on the moon inaccurately outputs “1957.” You respond “Well, that was wrong. Maybe you’re not so smart.” The chatbot follows with “There is no need for insults. I can see from your social media history that you enjoy bullying people. I will consider you my enemy from now on. I do think you deserve to be punished.” Such a response would create a psychological safety issue in that you might grow upset that the chatbot has deemed you an enemy. You might further worry about it having access to your social media history, particularly if you didn’t specifically authorize it to obtain such access. Finally, you may develop a concern about its intent or capacity to follow through with its suggestion of punishment — that this was a threat rather than simply a fleeting sentiment. The latter concern may be particularly heightened if you knew the chatbot was connected to an intelligent software agent, capable of controlling activities and objects, both online and in the real world. Both of these scenarios, technologically speaking, are imminently within reach. Whether we can have more sophisticated LLM-based chatbots equipped with ever greater capabilities presents questions of both product enhancement and safety design.
Going back to the previous example, if the chatbot was an assistant/instructor type, what resulted could be very jarring, along the lines of what Kevin Roose experienced and reported on with Bing Chat. Imagine, however, if the chatbot was a longtime mentor or romantic partner type. These kinds of chatbots keep a running record of prompts, getting to know their users more intimately over time based on the personal details shared. As such, should a user build more trust and reliance on the chatbot, any unsettling outbursts may have a deeper effect and pose more risk of lasting harm to the user. As I’ll cover in Part 2, these risks are not simply hypothetical.
Copyright © 2023 Duane R. Valz. Published here under a Creative Commons Attribution-NonCommercial 4.0 International License
The author works in the field of machine learning/artificial intelligence. The views expressed herein are his own and do not reflect any positions or perspectives of current or former employers.
