Towards GPT Design Patterns
Design Patterns and Heuristics for GPT and LLM AIs
If you are a UX designer or any related interaction designer, GPT and other chat AIs might strike you as being somewhat out of your bailiwick. What’s to design in a text prompt? It would seem that design is limited to interfaces that use AI to enhance some other application, such as image-making, writing, summarizing, etc. There the UX serves to enable functions, and design patterns are familiar.
But the prompt itself is a design space. It uses natural language, and accepts limited markup, but is nonetheless or even because of this a very open design space.
Does AI need UX?
UX doesn’t ordinarily consider language as a design domain. That’s left to NLP, SEO, perhaps some taxonomists and information architects (think categories, menus, classifiers, meta data, relationships, etc), and the like. Those would be the design of information organization.
But there’s an interaction component to language where language is communication and not just information content. As prompt engineering is a means of instructing GPT and other AIs to generate responses, there’s an interaction aspect to prompt engineering.
Most of the prompt engineering being shared to date involves providing AI with greater specificity around desired outputs. Prompts are crafted to guide the AI to a particular topic, person, story, or event, for example. They are used to request a response in a desired style or voice. Perhaps they ask for an impersonation, or for the AI to behave in character. These are all ways of using prompts to obtain a kind of conversation and interaction.
Prompt Engineering and Social Interaction Design
From my perspective, which is Social Interaction Design (SxD), then, these are a variation of interaction design. The user is engaged in an interaction with the AI, has a desired interaction and experience in mind, seeks a particular outcome or result, has an objective (usually), and has learned the AI sufficiently to reflect on how to frame prompts, instructions, requests, questions, and other conversational forms to achieve them.
Many of the attributes of a real-world social interaction apply. In fact the AI works, and the user is able to use it, without once referring to a manual of operation, precisely because it uses language. We can say that the user knows how to use the AI because both the user and the AI “know” something about communication. Namely, they know both how to talk and how to write.
GPT: Speaking and Writing
There is more then to the prompt than just using language. GPT and other language-based AIs can both converse and write. From a design perspective, this represents the first bifurcation: speaking and writing are two modes of the use of language. We will see further variations on speech in a moment. For writing, we have a wide range of types of writing, most of which are self-explanatory, though some of which are surprisingly effective: documentation, lists, code, structured tables, classifications and classifiers, emails and correspondence, and so on.
There are design requirements for the presentation of written GPT output, but I won’t go into those in depth here. We can simply state that GPT’s written responses have a language, form of content, a type of content, and possibly a use of structure. Functional code employs a coding language, structured, and deployed within a coding environment. Legal documents employ a conventional form, are a type of legal document, and are structured to the degree they are designed to be easily digitized, etc.
For the moment we can say that it appears that GPT can produce existing written responses in a variety of forms as long as they are standardized. We can only assume that new GPT-specific forms will emerge to serve purposes made possible by image, audio, music, video and other creative and editing AIs.
GPT: Conversation
It is in the conversational mode of GPT that interaction design comes into play. GPT’s facility for speech (in text form), its conversationality, its ability to go several rounds and retain and build on what has been “discussed,” are often the reasons we describe it as magical. Magical, because it is natural.
So you might ask: What is there to design in a natural language text prompt? The design is in language and its uses. Prompt engineering is not only engineering. It is also interaction design. The quality of the user’s experience, the validity and successfulness of a query and response, the user’s ongoing use of GPT, the branding of the AI’s personality and style of conversation—these and more are familiar design attributes to interaction designers. And they’re achieved with a thoughtful approach to prompting. Prompts are usually straightforward sentence and phrases. But they can be structured as well (see examples in the referenced Prompt Pattern Catalog paper). And this structure could be organized in the way that design patterns might be organized.
Branded AI
I foresee these patterns as being interesting because fine-tuned, customized, and branded chat AIs are going to become a commonplace. And the companies that provide them to customers and users will need to control their knowledge, behavior, and personality. These will be as much “brand experiences” as other facets of a brand. And so, I think interaction designers might help to formalize these AI interactions through co-designing the AI in training, and architecting prompts for use by end-users.
AI Personas and Personalities
Consider an obvious application of prompt engineering: personas. Branded AIs will have personalities. These might be based on existing and historical brand mascots. They might be based on current brand ambassadors, such as celebrities or other influencers. They might be real, or imaginary. Regardless, personality is a convenient, natural, and flexible way of constraining conversational behavior. Personalities could be designed to be knowledgeable in the brand product suite, the customer lifecycle (from discovery to servicing), product updates, upgrades, pricing, bundles, and so on. They could be designed to be friendly, snarky, complimentary, informative, educational, or helpful.
Personalities might be given histories or biographies. They could be trained on easter egg-like texts for customers to discover. They might be given event schedules, favorite television shows, local or regional histories, and a lot more. A lot more than a persona, in fact. All these and more could be designed, with the brand in mind, for training the AI.
Games with AI
Where personalities might capture the AI’s identity, interaction designers can also define ways of interacting. Obvious options for this would be games. There’s no reason that an AI should only take prompts or queries from users. They can be designed to reverse the roles, and for the AI to prompt users instead. (In this prompt pattern whitepaper it’s called flipping the script. I prefer reversing, but in social interactions it’s simply a form of turn-taking.)
AIs could be designed to give interviews; to ask questions as a teacher or even a therapist might. Questions can be intended to drill down into a topic, or to clarify intentions, needs, requirements, context, and so on. There are many more games that can be designed, as language games, or as gamified interactions: imagine an AI that gave or took away points from users as rewards and punishments; guessing games; word games; games with multiple rounds or never-ending games.
AI with Attitude
Tone and attitude are also design-worthy options for GPT-like chat AIs. Conversational tone might be associated with a personality but is in fact different. Where a personality represents an identity, tone and attitude describe momentary runs of conversation. Friendly, helpful, encouraging, for example. Or for a coach: challenging, encouraging, competitive. These would be embedded in a branded or customized version of an AI by means of fine-tuning on sample text. At some future point this might include sample interactions or conversations.
It would not surprise me if brands ultimately employ dramatists, psychologists, and other performance-minded consultants to help shape these conversational AI brand ambassadors.
User personalization
Eventually, when technology makes it feasible, chat AIs will be personalized. At this point interaction design and user experience will be required to help users negotiate the transparency of personal data and activities, scope of interactions and use cases, notifications, nudges, and so much more that AIs will be capable of using in their capacity as personal assistants. Some personalization will be deployed “at the edge,” on mobile devices, on proprietary chips, and with protected private data. This will give the AIs a memory they currently lack, and with memory, they will be able to engage in tasks now outside their scope: predictive analytics, locations, settings and preferences, account logins, notifications, etc.
AI and Memory
A common complaint from users of GPT is that it sometimes curtails access to past conversations. Or limits the number of prompts stored in a series. We assume that this has to do with minimizing GPU use and thus is cost-savings. But it’s also been noted that LLMs drift with the number of interactions sustained in a given use case. Temporality, in the forms of memory, recollection, and duration, present areas for AI design.
This topic is large enough as to warrant a discussion all unto itself. How long should a conversation thread run? How long should an AI’s memory last? Should each interaction, as a series of prompts and responses, be held distinct from all others? Or might there be strings of prompt series (grouped by their common topical association?). Keeping past interactions in memory (on file, accessible) could enable forecasting or anticipating user behavior. And access to past user behaviors would be essential to personalization. These will be longer in development, as memory and processing constraints, as well as privacy concerns, are development hurdles.
AIs will seem to evolve significantly as they hold more identity over time, and as they appear to remember users for longer, and across more and distinct interactions. And it is likely that use of memory, retaining past prompts and interactions (the prompt, response, and feedback on the response, for example), and any topical narrowing, expansion, drift, etc will also shape LLM model development. In fact the AI will appear to have more memory as it also has more access to personal user data.
Prompting Incentives
Rewards and incentives are not yet a facet of LLM AI interactions, but could be. Fortunately for us, GPT and other LLM AIs are not yet tied to algorithmic incentive and feedback models, such as those used in social media, news feeds, social networking follower models, trending, recommendations, and the like. But imagine that they could be.
Not only could GPT et al be integrated into these social incentive systems, GPT itself could be given incentives. It could be made to reward prompts with tokens or points. It could be made to encourage users to try again, clarify a question, dig deeper, and reward them for doing so.
By rewarding user engagement GPT could advance its own reinforcement learning and also tune its own user data. Prompt engineering can be bi-directional. Incentives could be used to shape and influence user prompting.
Formalizing Prompt Patterns with Interaction Design
The above observations barely scratch the surface of the formalization opportunity in prompt engineering, from a user interaction perspective. (I have not touched on recommendations, context, popular/trending, multi-agent interactions, and other possible features of prompt formalization.) There are already formalizations of prompting that serve to codify prompting for code, actions, structured data presentation, api calls and so on.
I am curious to see how the Interaction Design and User Experience communities take this on. I don’t think prompt engineering belongs entirely to developers, and do think that perspectives from user experience design warrant consideration. And I think it only natural that UX and IxD could develop a design pattern language or catalog with which to clarify thinking about the design of LLMs.
I have an initial and very preliminary pass at terminology and attributes captured in a matrix for prompt organization. But it needs to be mapped to prompts, context, examples, and much more, before it is useful.
The work would involve structuring and organizing approaches to prompt design, development of approaches to games and personas, conversational heuristics (how to talk to AI), and exploration of longer-term areas of development, such as memory, personalization, incentives, etc.
I’m curious, as always, to know whether others in UX and IxD find this to be a valid enterprise.
