Answering your design questions

Q&A with Voice AI designer Elaine Anzaldo and Wearables designer Jenny Lin

elaineinthebay
Bootcamp
Published in
18 min readDec 29, 2024

--

cover image of a photograph with a pothos plant in a glass vase with title, “design Q&A”, overlayed. to the side of the title, there are 2 instagram mock question templates with example questions for the article.
Cover image courtesy of Kulbir on Pexels

Hi readers, this one’s a bit personal. I’ve purposefully organized this design Q&A during a season in my career where I’ve lost a little of the magic and thrill of the craft. Every job, every job has highs and lows. I’ve learned that the more we ignore the lows, the harder it is to move past them. So, I’m writing to remind myself of why I got into this field in the first place. Call it what you want: conversation design or AI design. We have such a cool job :)

At the start of December, I asked my instagram followers to send me any questions under the sun related to: conversation, voice, AI, or wearables design. Yes, I know this Q&A format works better in a youtube video, but I couldn’t help myself. As much as I enjoy public speaking, I’m a much better writer. And if you’re new here, let’s just say I’ve had a realization over the past year: a lot of what I do can be described as conversation design, but that title no longer encapsulates everything I do. I hope this article can bring more awareness to what designing for AI is like, voice or otherwise.

Before you dig into the rest of the article, I wanted to acknowledge my Wearables Design coworker and literal girlboss Jenny Lin, who graciously accepted to be interviewed for one of the topics down below. I’m forever grateful to my job for the chance to work alongside top tier design talent. They’re the people I see present in design crit with impeccable storytelling, design craft, and, most importantly, grit. Jenny is one of them. Get ready: there’s a lot of resources and tips in this one!

Questions from the audience

I’ve sorted everyone’s questions into 5 different categories: Foundations, Craft, Corporate, Career, and Fun. Jenny’s career story is further ahead in the next section. If you like this article, please leave claps or comments, so I know to do another one. As always, happy reading!

Foundations

Question: How to get started with voice/IVR design?

I’d like to say that voice is just another user interface, but there are, in fact, as many unique challenges with the technology itself as there are with the social implications of the modality. When we design for voice, the goal is to make the exchange feel as “natural” as possible. To do this, you will need to understand the basic principles of human communication even before you map out your first user journey.

Communication “rules” and when to break them

Humans talk… a lot. We talk to exchange information, ideas, plans— we talk to give or receive companionship, understanding, moral support, and so many other things. There’s a lot embedded in what we say, whether we notice it or not. Things like our upbringing, cultural identities, or beliefs can be revealed through a simple exchange. Some might say this is what makes us “wired for speech.”

At its core, communication is a two-way street, and how we talk is as important as what we say. The “how” part is interesting because for a conversation to go smoothly, both parties must follow some kind of unspoken “rules” of conversation. One daring linguist defined these rules, which we now know as “Grice’s Maxims” (thanks Paul Grice!). The funny thing about conversation rules though, is that they’re kind of more like suggestions. For example, when I go home and tell my husband about Gossip Girl fan trivia even though I know he doesn’t care about the show, I’m not actually being very cooperative in the Grice sense.

As Brielle Nickoloff wrote in her piece, “The last Grice’s Maxims article you’ll ever need to read,” (which you should definitely read if you haven’t already): “a good designer is able to decide whether or not their design observes, flouts, or violates a maxim based on the specific context of the conversational exchange, and whether that observation, flout, or violation is a good thing or a bad thing in the eyes of the user.” As a voice designer, the onus is on you to make sure the voice experience sounds like a natural conversation by auditing how your users will react to it.

Technical considerations (why voice AI is so hard)

Voice technology has a lot under the hood. When you interact with a voice AI system, you’re encountering much more than simply the AI response. Anything you say first has to get converted from human speech to machine recognizable input— this then gets processed and matched to a use case, from there, the system needs to retrieve the knowledge associated with that use case, fetch data, retrieve response content, format it, and finally, all of that needs to get converted back into human recognizable output. I call it the “listen, think, respond” process because that’s literally what machines have to do to keep up with us.

infographic of the “listen, think, respond” process in conversational AI.
snapshot from my 2023 IxDF masterclass

All this to say: there are many, many points during this process where something might go wrong. One step could malfunction and ruin the entire user interaction. For example, if our ASR model is not performing well at the very front of the entire process, it doesn’t matter how great our NLU is or how accurate our LLM response is because the query will get misrouted and the user might experience an unexpected error because of it. It’s a lot like metabolic pathways in biology— if enzymes or chemical reactions don’t work as intended, our bodies can suffer as a result.

Why is this important? Voice designers might be expected to assist their teams in triage. Since you’re the expert on how the AI should behave, your team may look to you as the source of truth to help troubleshoot issues. Overall, it’s important to know your tech stack, not only to understand what’s possible and what isn’t in the design process, but also to audit the performance of your design in live traffic, identifying what is the limit of your technology versus what is a bug.

Design considerations (why voice is not for everyone)

Unlike traditional graphic user interfaces (GUIs) which are largely static and deterministic, voice UIs are dynamic and probabilistic. Voice technology runs on statistical probabilities, which means we’re never 100% sure that something is going to work 100% of the time. This might be a little frustrating to grapple with at first, “hey, that’s not what I designed, why is the AI doing that?” but it’s the reality of working on AI.

There are many complexities in designing for voice, some of which are:

  • Voice UIs are ephemeral (exchanges are fleeting and rely on the user’s memory to be successful).
  • Voice UIs are not “scannable” (there’s no way to show off all of your features other than explicitly telling the user about them).
  • Voice UI progression is serial (humans can only listen to one voice stream at a time, so the user and the AI need to give each other room to finish speaking before continuing).
  • Good sound quality is required (responses can’t only sound good on paper, they need to be audible and clear in the device they’re streaming from— call audio, over-the-ear speakers, phone speakers, etc).

Language also changes over time. Ten years ago, we weren’t going around the internet saying “very demure, very mindful”. Ten years ago, people’s exposure of voice technology was limited to Siri, Alexa, Google Assistant, and everyone’s favorite IVR systems. Being a voice designer means being diligent about keeping with up cultural changes (language evolution) and technological advances.

Advice for brave new voice designers

Personally, as someone who works on a displayless wearable device with voice as a key input, I’d recommend for voice designers to:

  • Hear what your AI responses sound like OUT LOUD 🔊 (and optimize your content for speech; that means no emojis, y’all)
  • Understand your tech constraints related to sound, i.e. your ASR model and Speech Synthesis system.
  • Learn how to describe sounds and how to collaborate with audio producers, sound designers, or voice actors.
  • Develop intuition for human conversation, knowing which aspects to apply to your designs and which to leave out.

If you made it this far, I’m proud of you, but the journey doesn’t end here! Throughout your career, your biggest asset will be your willingness to try voice products. This means getting very comfortable testing and reviewing the voice AIs of the world. No one embodies this spirit better than my conversation design colleague, Katy Boungard. Back in the day, Katy used to play Alexa games on her twitch channel. Nearly all of what she said back then holds true today. Please, if I can leave you with any wisdom today, it’s this: try out voice experiences! You cannot design for voice without ever experiencing voice. Make it a habit to observe what feels good and what doesn’t— you’ll rely on these skills repeatedly.

List of resources:

TL;DR every voice designer should know: communication principles, including cognitive biases like the primacy and recency effects (the first and last few items of a list are easy to remember), sound design basics, their tech stack, competitor experiences, and language usage* in their target market(s).

*Note: noticing language patterns and usage does not require native level proficiency of the language. In fact, if you are someone who knows another language aside from the one you design in, this is your superpower. You can help inform designs on whether they’re inclusive or confusing to a wide range of users.

Question: A lot of designers are trying to get up to speed with AI tech, from increasing productivity to designing AI products that are user-centric. What are the best resources today? What is the design process for designing AI products?

I will start off by saying: AI experiences do not work well without user trust. If you’re getting into AI design, it’s important to focus on topics like data transparency, ethics, and privacy and hard skills like AI prototyping.

These are the resources I would recommend checking out to help get into the mindset of AI product design, which is like HCI-informed guessing at scale. Note: this list is a starting point and is not comprehensive.

As for the design process (which I totally get that from the outside it looks a bit mysterious), I find myself relating very much to this diagram by Tetiana Sydorenko in her own Medium post, “AI product design: Identifying skills gaps and how to close them.” There’s so many shortcomings with the double diamond model when applied to AI design.

challenges in designing AI systems, under each of the 4 sections of the double diamond design process, there are descriptions of gaps in each section. for example, under “Discover”, one gap is, “difficult to articulate what AI can or cannot do.”

Additionally, the traditional product design process doesn’t makes sense for AI design because another fundamental piece of the AI puzzle is missing: data collection. It’s ideal if an AI designer can not only define the expected AI model behavior and output (which is done by showing “examples” to the model), but also influence how the model is trained. In conversation design, this step has usually taken the form of guidelines: very specific written instructions for how human workers are to grade AI responses or annotate (prepare) data to be fed into the model.

Designing for AI inherently means thinking in systems, patterns, and consequences. Designers can and should try to influence more than just AI response “content” or visual “templates.” Remember: AI has been designed and still needs designers :)

Question: Which books and online courses do you recommend for a UX Designer who wants to design Gen AI experiences and wants to know more about conversation design?

That’s so exciting! Aside from all of the resources I listed in the answer right above, if you want to get closer to conversational technologies, I would also suggest the following resources (CxD related):

cover thumbnail for the article, “the ultimate conversation design course guide” by elaineinthebay, author of this article.

And if that’s not enough, I wrote up an entire article with a list of course options for every budget. Also includes student testimonies!

Craft

Question: Do conversation designers decide whether to use LLMs vs NLU vs hybrid for a product?

Theoretically, yes, senior conversation designers should participate in the discussion by grounding their team in user research (asking questions like: what do users actually need solved?). Or, in cases where CxDs do account management, they should help guide their clients in the best tech solution for their specific needs.

Realistically, no, conversation designers are typically the last to be hired onto a Conversational AI product team, which means other stakeholders are usually the ones who decide the technical approach.

Question: What’s the most surprising thing about designing for wearables?

I was surprised by how much style matters. Obviously, wearables are called as such because users are meant to wear them on their body, but I guess I didn’t realize how much fashion, comfort, and fit matter to the everyday consumer. I know people who’ve purchased Ray-Ban Meta for the looks and only realized after purchasing that it came with other features like audio streaming, calling and messaging, multimodal AI, etc.

Question: What is the hardest part of the CxD job?

The hardest part of the job (job, not career) is stakeholder management. Pre-LLMs, our job was to oversell our technology and convince our cross-functional partners and clients that AI could do more than just answer FAQs. Now, our focus has shifted from offensive to defensive. Post-LLMs, we mostly try to undersell our technology and temper expectations. Many of our partners and clients are assuming their product requirements can be fulfilled by LLMs out of the box, but our role is to highlight the work still needed: where design can drive decisions, add guardrails, and adapt the experience to user feedback.

Doing all this is not always easy, especially when the topic is natural language because everyone and their mother assumes they’re an expert in it by default (which they kind of are, but not in the CxD sense). Demonstrating your expertise in human communication can be a battle on its own.

Career

Question: How can new grads learn about AI design and relevant skills for better job opportunities?

Audit AI products, not just once, but often. In an old interview about conversation design, Hilary Hayes described her personal process where she’d keep notebooks next to each of her smart home devices to jot down responses, behaviors, and overall observations about the product. Doing so allowed her to see how these voice experiences evolved over time, instead of observing them only in a specific moment in time. Competitive analysis is one of the top skills for anyone working in AI, particularly since the space moves fast. I’d recommend honing this skill above others and demonstrating it actively throughout your portfolio.

Question: How did you prep for your job search when you were looking for a job?

You should probably know that the last time I was interviewing (in 2022), I wasn’t even expecting to land an offer. You know that advice: always take the interview, even if you aren’t looking? That’s what I did. Around 1 year into my first full-time role in design, I wanted to see what was out there.

What happened was this: several recruiters found me and messaged me directly. I spoke to them, interviewed with the team I liked and didn’t prepare much at all. At the same time, a friend of mine told me about an opening on her team at Meta and asked if I wanted a referral— I said yes. I tried to be really casual about it but actually I was ecstatic and freaking out. I hadn’t interviewed at a FAANG before, so I felt like the odds were stacked against me and I had to overcompensate for my lack of pedigree.

The interview process consisted of: 1 recruiter screen, 2 panel portfolio presentations, 1 app critique, 1 whiteboarding session, 1 casual 1:1 with another designer on the team, and 1 hiring manager chat. I was pretty good at 1:1 interviews, so I didn’t prep for them, but the rest? It was my first app critique and my first whiteboarding exercise so I tried to study as much as possible before my interview dates were scheduled.

screenshot of instagram story from Feb 28, 2022.
snapshot of my instagram story the night before my Meta interviews

Here are the resources that helped me the most:

I should also mention my recruiter was a huge help during the interview process. Meta recruiters are an absolute gem— they genuinely care about their candidates and provide lots of feedback to help you succeed. My recruiter was extremely proactive and gave me actionable tips on how to improve my portfolio deck, things like reducing the amount of text per slide and adding more design iterations into my case study.

The other thing I did passively (that is, did not prep explicitly for the interview) was create my personal brand. I truly believe the thing that made me appeal to the team was my unique voice, which I had carefully crafted for over a year before the interview. I mentioned this in another article: “[M]y personal brand also came into play during the interview process. I stood out because of the work I had put into showcasing my thought process and passion for Conversational AI in my blogs! Someone in my interviews actually quoted me back to me — it was amazing!

Question: Resources to prep for conversation design or chatbot design interviews?

It’s been a while since I’ve had to interview for CxD roles, so the only advice I can give is this: above all else, nurture your communication skills. Market yourself hard in your interviews. If you can’t sell yourself, the hiring team won’t believe in your ability to sell your designs. Why you? How are you an asset to the team? How can you save the company time or money?

For tangible resources, I highly recommend checking out: my “How to get hired in CxD” series, parts [1], [2], and [3], as well as Hillary Black’s remix, Convofolios, which is my collection of CxD portfolio examples, and my evergreen resource, the “Conversation Design Job Interview”.

Corporate

Question: Does it feel like conversation designers and other designers are valued the same at your company?

A spicy question deserves a spicy answer: no, it doesn’t feel like we’re valued the same as other designers. It feels like product designers reign supreme and the rest are kind of nice-to-haves. If I didn’t know any better, I’d say yes, we’re all equal (at least on the HR level we are). Alas, since I got to experience a culture that did value CxDs once, I can’t say the same for this one. Of course, that’s not to say product or content designers don’t have their own org struggles to deal with and, generally, things are not as bad as they might sound. Honestly, I’m proud of my team and of the work we’ve done. It’s just… I’m so done with feeling like a minority in a huge design org. Hopefully some of the upcoming changes will level the playing field.

Question: Do you actually like working at a FAANG company? Do you think you will again one day?

I actually love it. Much of my distress from working at a FAANG company has nothing to do with the company and more to do with the everyday grind. Look, I come from an unglamorous background in tech. My biology degree didn’t get me any jobs here, so I worked contract jobs (bottom of the barrel jobs) for nearly 3 years to gain experience. I didn’t personally enjoy working through illnesses, filling out timesheets, and wondering every day if I had a job to come back to the next day. I appreciate the relative job stability I have by working at a huge tech company.

Would I like to work at a FAANG again someday? Yes, 1000%. Preferably the same company. I’ve gotten so used to this company culture that I can’t even imagine working for a competitor.

Question: What do you wish other coworkers knew about how to best work with conversation designers?

The first thing I try to emphasize when working with someone new is: CxD doesn’t mean string writer (that’s why our job isn’t obsolete in the age of Gen AI). Just how product designers cannot be simplified to “pushing pixels”, conversation designers can’t be reduced to “writing dialogs”. We’re responsible for a multitude of complex interactive tasks, such as: defining product requirements, determining interaction logic and system response formatting, refining model training, and absorbing user needs to create solutions that incorporate tech constraints as well as business needs.

Fun

Question: What do you use your Meta Ray-Bans for?

First off: fashion (obviously). RBMs are the fanciest accessory I’ve ever bought for myself. Before joining this team, I used to go around wearing my free pair of plastic sunnies I got at some university fair. Even now, I feel like a movie star when I wear them.

Usually though, I’m wearing my RBMs to quickly take photos and videos throughout my day. I’m not super particular with my photography style, but I am super adamant about documenting the small things that bring me joy throughout the day. Afterward, I might upload some of those captures to my IG story. I use them while driving— e.g. to message my spouse while I’m out running errands— and I’m using the AI features, either to test things for work or to identify flora and fauna. The first time I used multimodal AI was to settle a quick question between me and my husband: were we actually looking at a Redwood tree? The answer was yes :)

Question: Biggest pet peeve as a CxD

My absolute biggest pet peeve is when people refer to conversation design as ‘conversational design’??? I don’t get it.

Designing your career: chat with Jenny Lin

For the following questions, I decided to join forces with a dear colleague of mine. To answer, “as a junior designer, what’s the best way to get into wearables?”, I sat down with senior product designer Jenny Lin to learn from her journey into wearables! The full interview is on youtube. In it, we discussed how she shifted from having a degree in advertising and working on monetization to landing a job in AR amidst obstacles and rejection.

We also answered the question, “How would you recommend getting a portfolio on wearables?” You can check Jenny’s portfolio here. The gist of it is: there’s no set formula on how to make a portfolio that will guarantee you a job in the wearables space, but it’s important to show a) your passion for emerging tech in your portfolio and b) your take on existing wearables products in the market.

Even if you do not own a wearables device yourself, there’s still a way to gain that experience. Jenny says, “There’s a lot of AI apps that have voice modality […] play around with them. Imagine the form factor is not from a phone […] maybe it’s a watch, maybe it’s glasses, out in the market, there’s also a ring. Imagine what pains the user will have, what the user journey will be, what problems will be worth solving or that are not being solved currently.”

Imagine what user problems are not yet solved with the products of today.

During our conversation, Jenny also mentioned, “Product design is mainly about problem solving […] the main difference between wearables and traditional product design is the modality, the input modality, the output modality, but the essence doesn’t change. It’s still solving problems.”

If you are looking to get into wearables design, don’t give up! If you really want to get into the space, you will— you just have to keep trying.

--

--

Bootcamp
Bootcamp

Published in Bootcamp

From idea to product, one lesson at a time. Bootcamp is a collection of resources and opinion pieces about UX, UI, and Product. To submit your story: https://tinyurl.com/bootspub1

elaineinthebay
elaineinthebay

Written by elaineinthebay

AI Designer✨ | Voice, Product, & Conversation Design

No responses yet