UX Prototyping for AI

Published in

Salesforce Designer

9 min readAug 9, 2021

In human-centered design and agile software development, it’s hard to overstate the importance of prototyping. Prototypes are incredible, low-investment tools for learning and validation. These preliminary experiences are necessary when it comes to AI given its enormous overhead (e.g. data collection, model building, reliability testing and maintenance). So it makes a lot of sense to prototype the product ahead of time.

But how can UX teams prototype AI-powered experiences?

That’s exactly what our team asked while creating Einstein Designer. Unlike traditional software tools that can be tested with a linear click-prototype, AI tools are frequently open ended. For example: While a file-upload is either uploaded or not, a chat bot or voice assistant needs flexible response options.

In this article, I’ll discuss six different prototyping methods we employed.

AI Prototyping Phases

In any AI project there are two major phases (which could happen concurrently): product vision and model development. For each of these two phases there are two broad categories of prototypes: value prototypes and interaction prototypes.

In the product vision phase, prototypes help answer the questions: what product should we make, and how should humans interact with the AI? Prototypes don’t directly help answer this question, but they do help you test your hypotheses. As discussed above, prototypes provide a cheaper alternative to building the full product, allowing your team to explore multiple options and get feedback from stakeholders and users about the potential of each. Answering these questions for the model itself is very similar but more granular. Of all the things a system can do, not all of them will be predictions. Model value prototypes can help you identify the key areas where you need AI vs where a heuristic or a human would be better.

When it comes to interaction, prototypes can help you understand the possible inputs and how people want to provide them. You can learn how the system should work. The challenge with AI is that how it should work isn’t always the same as how it can work. It’s so easy to design for magic, but some prototypes can help keep our imaginations grounded in reality.

Types of AI Prototypes

Some are automated such as Input/Output prototypes or Heuristic Systems. Others, like what I call Wizard of Oz or Humans as AI, involve human support. Then there are niche prototypes — including Data Visualization/Storytelling and Constrained Scenarios, which are for specific project needs. Let’s review them all and how our team engaged with each.

Input and Output

This is one of the most unique prototypes to AI projects. It is also one of the cheapest. In this kind of prototyping, you create samples of inputs and outputs as a means of testing the value of this service.

Our team did this when we drew samples of design versions we hoped Einstein Designer would generate given an input of some text and an image. To execute this, we simply put them on a slide like the one below and asked customers to give us feedback on how useful they were. This simple method allowed us to learn about how customers consumed, evaluated and compared designs.

Image of design slide prototype showing user feedback

Humans as AI

The wonderful thing about AI is that each of us is already in possession of the most powerful AI we know of: the human mind. Sure, we can’t do everything machines can (mainly due to scale and throughput issues). However, we can still use our own minds to simulate the output of a machine learning model. If we humans can’t make heads or tails of some inputs, then there is a pretty good chance that you’re going to have a hard time creating a prediction from the data. This allows you to validate whether or not a prediction is valuable if it were possible. Hopefully, you can find patterns and then codify those patterns in a model. This is, after all, what data scientists do every day.

We tapped humans as AI when we were exploring different ways of predicting font choices from a website. To rapidly experiment with which features were important, we put some sample data in Sketch (below) and asked team members what decisions would each make given the provided inputs. This allowed us to recognize very quickly that absolute font size and relative font scales are both very poor predictors. Instead, we learned we needed to predict some landmarks such as the baseline font size of the brand’s design system.

Image of NYTimes font prediction sketch file showing how different rules lead to different designs

Wizard of Oz

This prototyping method is similar to the previous one in that humans are used to simulate an AI model. The difference is that companies typically use this either to demo a vision of an experience or to do live user testing of the interactions. The key is that the humans are working live behind a curtain in order to create a responsive experience that appears to be fully functional. Imagine the scene in “Seinfeld” when Kramer pretends to be Moviefone. Other examples would include a remote human operator for either an automated car or automated chat bot.

On our project, we did some of this by building a code prototype using Chrome extensions. Then we pre-populated it with hand-curated results. The app seen below was used extensively internally to illustrate how the product would work before creating a model.

Image of chrome extension showing how the AI can provide alternative designs that are previewed in the UI

Data Visualization and Storytelling

While not strictly a prototyping method, these techniques do have a place in the AI prototyping tool kit. Just look at the breadth of sci-fi literature and media. While writers don’t actually build the futuristic robots, they are able to transport us to those futures and help us to explore what is possible, desirable and maybe even likely. A little closer to home is the work of Nick Foster with Design Fiction — ”a way to facilitate reliably productive conversations about possible products, services and experiences which yield more habitable near futures.” This can involve films, objects, websites, books or sketches.

Within our own project, we utilized the power of data visualization and storytelling to help explain the concept of Einstein Designer and how it works. This interactive data visualization — built in collaboration with the data viz wizard Moritz Stefaner — was used at the main Dreamforce keynote to announce Einstein Designer.

Image of particle animation at Dreamforce ‘19

Heuristic Systems

Heuristic systems are perhaps the closest thing to what you might normally imagine as a software prototype. They are also on the more expensive side of prototypes to make. These prototypes are built in code. However, instead of employing models, they use a set of heuristic rules that approximate the model output.

We used these extensively while building Einstein Designer. We built a series of design generators like the one below that tried out various rule-based methods for predicting font styles, layout and even placement. This allowed us to test our UX in a very realistic fashion where users were allowed to upload any image and enter arbitrary text. Through this process, we learned that some of our design-exploration interactions were very hard for users to understand. In later versions we focused more on presenting good alternatives out of the gate and less on having the user navigate the space of designs.

Image of Chrome extension showing heuristic based generation based on slider input

Image of app using heuristic design generation

Constrained Scenarios

Lastly, I wanted to mention a technique I learned about at a talk by some of the designers behind the “smart-compose” feature in Gmail (below). This technique is closely related to heuristic systems but helps limit the complexity. In their case, they wanted to prototype what it would be like for the user to have real phrases suggested and to try different keyboard interactions. To do this, they needed a real code prototype but obviously couldn’t build the whole model just yet. So they selected a very narrow scenario: the birthday email. This allowed them to solve the problem with brute force. A long list was created of all the possible birthday sentences and phrases a user might enter and they did a simple text comparison to suggest the right response.

While I didn’t get to use this prototype, what I love about it is the clever way the team constrained the problem so that they could rapidly prototype a solution.

Challenges of AI Prototypes

As these different methods illustrate, UX prototyping for AI presents some unique challenges that aren’t present in traditional software and services. I mentioned upfront how the fundamental open-ended format of AI is difficult because it requires handling many possible user inputs with an equally vast number of outputs. I also want to highlight additional bias-related challenge areas that shouldn’t be overlooked.

Predictions

How do you know how the system will actually respond? It’s not like a classic software product where you might create some sample data for your prototype. Instead, an AI system needs to generate plausible predictions. On the surface, this might seem easy. In fact, it can be so easy that you create predictions that aren’t actually representative of the model output. For this reason, it’s crucial to develop the prototype in tandem with engineers who can help guide the scenario in plausible directions.

For example, when drawing the prototype-generated designs, we were careful to highlight which properties were modified in Sketch. We took the fonts, font-sizes, colors, and kerning etc directly from the brand’s site. We then worked only with layout and alignment to create the designs. While we could have taken all kinds of liberties that are easy for humans to do, we didn’t because it was important to explore what the likely model output would be.

Ethics

Finally, we can’t forget to think about ethics. Prototypes don’t have the model’s biases, so we need to explicitly have conversations about model bias and unintended consequences of a product. Prototypes also don’t explicitly reveal the potential negative consequences of a product. However, they can help shape the conversation about negative consequences. Storytelling can be a useful tool to directly explore these consequences. Try having team members write, draw or describe possible scenarios.

Prototypes are also great for setting context and aligning participants in workshops around an idea. Having been primed, using a workshop methodology like consequence scanning can be a great way to talk about ethics. Be sure to include activities that help people consider a wide range of possible outcomes. After all, no one expected Alpha Go to win against Lee Sedol in 2016 but it did. The future has a tendency to sneak up on us.

Check out the Ethics by Design module on Trailhead

Learn about ethical design and how to incorporate it into technology development.

trailhead.salesforce.com

Now you know more than a few ways that UX teams can prototype AI-powered experiences.

See what works best for your project and what fidelity meets your needs. Choose from automated or human-supported types, including Input and Output; Humans as AI; Wizard of Oz; Data Visualization and Storytelling; Heuristic Systems; and Constrained Scenarios. If you have alternatives that have worked for you, share in the comments. A little sharing can go a long way.