In design, storytelling has become a fashionable trend over the past decade with many speakers, writers, and websites professing the importance of storytelling to design. While storytelling can be useful, it also runs the risk of misleading teams if not done conscientiously. In this article I’ll discuss the role of storytelling in design and how to use it wisely.

In my experience, storytelling is used for three primary purposes in the design profession:

1. Real stories of people who might use the product or service.
   As Whitney Quesenbery describes in “Storytelling for User Experience”, these stories, as captured and retold through user research, build empathy and understanding for other people’s experiences to help ground and inspire the team.
2. Stories embedded into products and services.
   The idea here is that great design should tell a story to the user. This idea is often criticized, because the majority of cited examples, like a website using photography or a roller coaster ride, seem to lack narrative development.
3. Stories that describe how the experience could be.
   These are speculative stories about possible futures. When used casually they can allow teams to rapidly explore possible solutions in an empathetic way. When formalized they can help align groups of stakeholders around a vision. There are many nuanced forms of stories in this category including “scenarios” and “design fiction”.

This article focuses primarily on this third kind of storytelling.

Recognize the power of narrative

Speculative storytelling plays a central role when organizations make decisions about what to build next. It helps teams consider the question: “What vision should we pursue?” Design strategy is about identifying opportunities and framing solutions at a high level to get the commitment to design and build them at a low level. It’s common practice, in my experience, to wrap arguments about what to build within a story to give them emotional weight. While I can’t deny the effectiveness of this approach, I’ve long harbored a queasiness about focusing on the craft of storytelling.

Everyone loves to call out misleading statistics, but misleading stories are even more pernicious. The problem, as I’ve come to realize, is how to tell when a story is misleading? Unlike statistics and other scientific data, I don’t have a robust set of tools to see through the smoke and mirrors.

Stories, good stories, have a power that is gripping. They dive deep into our psyches and prove convincing even when they are complete fiction. While some of our most beloved stories are intentionally fiction, I wonder how to distinguish misleading tales from those that contain an essential truth.

Question the future

Within design, the stories we tell are about the future. They’re about how people will use and love some product, and how that product will result in increased profits. These stories are fictions, necessarily, since they take place in the future. They are also seen as one of the best ways to convey the vision for a product because they capture the human experience. They engage the audiences’ empathy for the protagonists. Yet, one runs the risk of telling a compelling tale that leads the group off of a cliff. Just because I can tell a compelling story about a future product doesn’t mean we should actually build that product. So how can we distinguish helpful stories from misleading ones?

When considering a story within the context of design strategy, I suggest the following starting points for evaluation:

1. Is the end user of the product portrayed realistically?
   Is the actual user or person who will be impacted accounted for? Do they have realistic motivations? Are there quotes from real people to support the story? It’s too easy to tell stories of profit and power when the only players are abstract business units or users who love the product. When you are forced to include the actual humans in the story you are forced to address humanity and all its complexity. Take a story about mobility for example. Volkswagen surveyed American SUV drivers and found that 87% of them plan to stick with SUVs for life. They concluded that “our current portfolio of SUVs aligns with what consumers are prioritizing”. Unfortunately, this simplistic conclusion detracts from the actual needs they found, “comfortable seating” and “safety”. Richly describing what these mean to people would uncover opportunities to invent new forms of transportation as Tesla and others have done. The survey alone is equivalent to focusing on the proverbial faster horse, when cars are just around the corner.
2. Are all the characters in the story accounted for?
   Has the story accounted for partners, employees, regulators, and competitors? Even user-centric stories often skip important characters and gloss over the potential pitfalls. As Porter has shown, competitors have plans of their own; ignore them at your own risk. Returning to our mobility story, as companies consider the shift to electric cars it’s important to factor in all the different stakeholders. For example, how might employees of car companies be impacted? Bloomberg Law reported, “One-quarter of auto assembly workers could be displaced if U.S. consumers embrace electric cars, according to the Congressional Research Service, given that electric models are simpler to put together than their gas-guzzling predecessors.”
3. What are the larger societal trends surrounding this narrative?
   Much like with statistics, context is key to understanding a story. Be skeptical of any story that doesn’t account for the larger societal trends surrounding the topic. For example, has our mobility story accounted for the chronic debt crisis? Some have speculated that car loan defaults could be the next financial crisis. Has the narrative accounted for the need to electrify our entire transportation sector and power it with renewable electricity? Has it accounted for the role of public transportation?
4. What are alternative narratives?
   While it’s best for authors to provide some alternative narratives of their own, savvy consumers should consider alternatives to the alternatives. As Peter Schwartz explains in “The Art of the Long View”, it’s impossible to predict the future, but if you can identify a strategy that works in a variety of future scenarios, then you have a solid plan. You’ll still have to make course adjustments, but you have also effectively hedged your bets. In this model, one good counter-narrative can change everything. For example, even in a future where all vehicles are powered by renewable electricity, the marketplace could be radically different depending on whether utility scale generation or micro-grids dominate. The likely answer is a bit of both.
5. How does this story fit your own agenda?
   Con artists will tell you that the best of them tell their victims what they wanted to hear. Does the story sound perfect? Is it everything you ever wanted? How would you feel if it turned out to be false? Especially in cases where you would be disappointed if the story turned out to be untrue, tread cautiously. Your own emotions may be misleading you to believe this story when you shouldn’t. With mobility, are you invested in oil companies or electric vehicle companies? Are you sentimental about cars? Are you feeling any strong emotions about electric cars?
6. Does the story include a range of consequences?
   As they say, “there is no free lunch”. There will always be a range of negative and positive consequences to any product. And there will be a range of uses by both good and bad actors. A realistic story about the future should account for this range. One technique that can help you develop an understanding of what consequences to look for is “consequence scanning”. Even in a future where all vehicles are powered by renewable energy, there will still be a range of consequences. According to the US Department of Energy, “EVs typically require less maintenance than conventional vehicles.” Will this hurt the job prospects for mechanics?

This is not a complete list by any means, and I would love to hear if you have methods, frameworks, or ideas for how to detect misleading stories.

Prioritize real human connection

Designers who talk about storytelling love to emphasize that stories are emotional. It is precisely this emotional power that worries me about stories. When we are feeling intense emotions, we are also least able to make rational decisions. We are least able to discern truth from fiction. By framing the problem around creating stories, there is a risk that we allow the best story to win, when in fact we should let the most accurate and truthful story win.

I can’t deny that humans love stories and that stories are a powerful method of communication, but I can warn against emphasizing stories and the quality of their telling. As Yowei Shaw of Invisibilia says, people are “weaponizing narrative” to advance their agendas precisely because “people feel defenseless against narrative”. In a sort of narrative cold war, “it’s like whoever is the best narrative master will win. Nothing we can do except do it better?”, says Shaw. Telling better stories does not lead to better designs and better outcomes. Better design leads to better outcomes.

This is why I appreciate the emphasis that Relationship Design puts on real people. It correctly focuses on these real people rather than on any one particular design methodology and techniques for being more compelling. The goal is to build great products that enhance the relationships between real people who stand at the center of the design process.

As designers, our role is not to tell stories but to use stories as one method of communication in the process of creating great products and services that bring those people together. By focusing on the experiences of these people, Relationship Design can help ensure that the stories we tell each other remain grounded in reality. And hopefully by thinking critically about those stories we can establish a common truth that leads to great outcomes.

Stories are powerful tools to convey ideas and create alignment. When told truthfully and with great humility they can be a force for good. Just as easily they can be twisted and tailored to suit any need. Be skeptical when someone starts spinning a yarn and remember to ask questions. Whether it’s in design or real life, we need to develop our skills for how to see through false narratives.

Learn More

Thinking Like the Machine: Using Metrics to Generate and Quantify UX Design

The Role of Design in Machine Learning

UX Prototyping for AI

Design Strategist Trailmix

Thanks

Follow us at @SalesforceUX

Learn more at design.salesforce.com

Additional Resources Regarding Stories and Truth

Lex Friedman podcast with Ronald Sullivan, “#170 The Ideal of Justice”

Tim Harford’s “The Data Detective” at The Cato Institute

Radiolab podcast “Deception”

Design Stories and Truth was originally published in Salesforce Designer on Medium, where people are continuing the conversation by highlighting and responding to this story.

In human-centered design and agile software development, it’s hard to overstate the importance of prototyping. Prototypes are incredible, low-investment tools for learning and validation. These preliminary experiences are necessary when it comes to AI given its enormous overhead (e.g. data collection, model building, reliability testing and maintenance). So it makes a lot of sense to prototype the product ahead of time.

But how can UX teams prototype AI-powered experiences?

That’s exactly what our team asked while creating Einstein Designer. Unlike traditional software tools that can be tested with a linear click-prototype, AI tools are frequently open ended. For example: While a file-upload is either uploaded or not, a chat bot or voice assistant needs flexible response options.

In this article, I’ll discuss six different prototyping methods we employed.

AI Prototyping Phases

In any AI project there are two major phases (which could happen concurrently): product vision and model development. For each of these two phases there are two broad categories of prototypes: value prototypes and interaction prototypes.

In the product vision phase, prototypes help answer the questions: what product should we make, and how should humans interact with the AI? Prototypes don’t directly help answer this question, but they do help you test your hypotheses. As discussed above, prototypes provide a cheaper alternative to building the full product, allowing your team to explore multiple options and get feedback from stakeholders and users about the potential of each. Answering these questions for the model itself is very similar but more granular. Of all the things a system can do, not all of them will be predictions. Model value prototypes can help you identify the key areas where you need AI vs where a heuristic or a human would be better.

When it comes to interaction, prototypes can help you understand the possible inputs and how people want to provide them. You can learn how the system should work. The challenge with AI is that how it should work isn’t always the same as how it can work. It’s so easy to design for magic, but some prototypes can help keep our imaginations grounded in reality.

Types of AI Prototypes

Some are automated such as Input/Output prototypes or Heuristic Systems. Others, like what I call Wizard of Oz or Humans as AI, involve human support. Then there are niche prototypes — including Data Visualization/Storytelling and Constrained Scenarios, which are for specific project needs. Let’s review them all and how our team engaged with each.

Input and Output

This is one of the most unique prototypes to AI projects. It is also one of the cheapest. In this kind of prototyping, you create samples of inputs and outputs as a means of testing the value of this service.

Our team did this when we drew samples of design versions we hoped Einstein Designer would generate given an input of some text and an image. To execute this, we simply put them on a slide like the one below and asked customers to give us feedback on how useful they were. This simple method allowed us to learn about how customers consumed, evaluated and compared designs.

Humans as AI

The wonderful thing about AI is that each of us is already in possession of the most powerful AI we know of: the human mind. Sure, we can’t do everything machines can (mainly due to scale and throughput issues). However, we can still use our own minds to simulate the output of a machine learning model. If we humans can’t make heads or tails of some inputs, then there is a pretty good chance that you’re going to have a hard time creating a prediction from the data. This allows you to validate whether or not a prediction is valuable if it were possible. Hopefully, you can find patterns and then codify those patterns in a model. This is, after all, what data scientists do every day.

We tapped humans as AI when we were exploring different ways of predicting font choices from a website. To rapidly experiment with which features were important, we put some sample data in Sketch (below) and asked team members what decisions would each make given the provided inputs. This allowed us to recognize very quickly that absolute font size and relative font scales are both very poor predictors. Instead, we learned we needed to predict some landmarks such as the baseline font size of the brand’s design system.

Wizard of Oz

This prototyping method is similar to the previous one in that humans are used to simulate an AI model. The difference is that companies typically use this either to demo a vision of an experience or to do live user testing of the interactions. The key is that the humans are working live behind a curtain in order to create a responsive experience that appears to be fully functional. Imagine the scene in “Seinfeld” when Kramer pretends to be Moviefone. Other examples would include a remote human operator for either an automated car or automated chat bot.

On our project, we did some of this by building a code prototype using Chrome extensions. Then we pre-populated it with hand-curated results. The app seen below was used extensively internally to illustrate how the product would work before creating a model.

Data Visualization and Storytelling

While not strictly a prototyping method, these techniques do have a place in the AI prototyping tool kit. Just look at the breadth of sci-fi literature and media. While writers don’t actually build the futuristic robots, they are able to transport us to those futures and help us to explore what is possible, desirable and maybe even likely. A little closer to home is the work of Nick Foster with Design Fiction — ”a way to facilitate reliably productive conversations about possible products, services and experiences which yield more habitable near futures.” This can involve films, objects, websites, books or sketches.

Within our own project, we utilized the power of data visualization and storytelling to help explain the concept of Einstein Designer and how it works. This interactive data visualization — built in collaboration with the data viz wizard Moritz Stefaner — was used at the main Dreamforce keynote to announce Einstein Designer.

Heuristic Systems

Heuristic systems are perhaps the closest thing to what you might normally imagine as a software prototype. They are also on the more expensive side of prototypes to make. These prototypes are built in code. However, instead of employing models, they use a set of heuristic rules that approximate the model output.

We used these extensively while building Einstein Designer. We built a series of design generators like the one below that tried out various rule-based methods for predicting font styles, layout and even placement. This allowed us to test our UX in a very realistic fashion where users were allowed to upload any image and enter arbitrary text. Through this process, we learned that some of our design-exploration interactions were very hard for users to understand. In later versions we focused more on presenting good alternatives out of the gate and less on having the user navigate the space of designs.

Constrained Scenarios

Lastly, I wanted to mention a technique I learned about at a talk by some of the designers behind the “smart-compose” feature in Gmail (below). This technique is closely related to heuristic systems but helps limit the complexity. In their case, they wanted to prototype what it would be like for the user to have real phrases suggested and to try different keyboard interactions. To do this, they needed a real code prototype but obviously couldn’t build the whole model just yet. So they selected a very narrow scenario: the birthday email. This allowed them to solve the problem with brute force. A long list was created of all the possible birthday sentences and phrases a user might enter and they did a simple text comparison to suggest the right response.

While I didn’t get to use this prototype, what I love about it is the clever way the team constrained the problem so that they could rapidly prototype a solution.

Challenges of AI Prototypes

As these different methods illustrate, UX prototyping for AI presents some unique challenges that aren’t present in traditional software and services. I mentioned upfront how the fundamental open-ended format of AI is difficult because it requires handling many possible user inputs with an equally vast number of outputs. I also want to highlight additional bias-related challenge areas that shouldn’t be overlooked.

Predictions

How do you know how the system will actually respond? It’s not like a classic software product where you might create some sample data for your prototype. Instead, an AI system needs to generate plausible predictions. On the surface, this might seem easy. In fact, it can be so easy that you create predictions that aren’t actually representative of the model output. For this reason, it’s crucial to develop the prototype in tandem with engineers who can help guide the scenario in plausible directions.

For example, when drawing the prototype-generated designs, we were careful to highlight which properties were modified in Sketch. We took the fonts, font-sizes, colors, and kerning etc directly from the brand’s site. We then worked only with layout and alignment to create the designs. While we could have taken all kinds of liberties that are easy for humans to do, we didn’t because it was important to explore what the likely model output would be.

Ethics

Finally, we can’t forget to think about ethics. Prototypes don’t have the model’s biases, so we need to explicitly have conversations about model bias and unintended consequences of a product. Prototypes also don’t explicitly reveal the potential negative consequences of a product. However, they can help shape the conversation about negative consequences. Storytelling can be a useful tool to directly explore these consequences. Try having team members write, draw or describe possible scenarios.

Prototypes are also great for setting context and aligning participants in workshops around an idea. Having been primed, using a workshop methodology like consequence scanning can be a great way to talk about ethics. Be sure to include activities that help people consider a wide range of possible outcomes. After all, no one expected Alpha Go to win against Lee Sedol in 2016 but it did. The future has a tendency to sneak up on us.

Ethics by Design

Now you know more than a few ways that UX teams can prototype AI-powered experiences.

See what works best for your project and what fidelity meets your needs. Choose from automated or human-supported types, including Input and Output; Humans as AI; Wizard of Oz; Data Visualization and Storytelling; Heuristic Systems; and Constrained Scenarios. If you have alternatives that have worked for you, share in the comments. A little sharing can go a long way.

Learn More

How to Build Ethics into AI — Part I

How to Build Ethics into AI — Part II

Three Fundamentals of Human-Centered Design in the Age of AI and Machine Learning

Einstein Designer — AI-Powered, Personalized Design at Scale

Thanks

Sönke Rohde, Jessica Lundin, Michael Sollami, Alan Ross, Brian Lonsdorf, David Woodward

Learn more about Salesforce Design at www.salesforce.com/design

Follow us at @SalesforceUX.

Check out the Salesforce Lightning Design System

UX Prototyping for AI was originally published in Salesforce Designer on Medium, where people are continuing the conversation by highlighting and responding to this story.

Sample Machine Learning: Redrawing the Innovation Roadmap

Machine learning (ML) technology has the possibility of fundamentally changing how we approach innovation.

Instead of the traditional moonshot approach (iteratively building/improving features toward a lofty goal), our team has had success using a data-first approach* to meet monumental benchmarks. We honed this direction while helping create Einstein Designer, an AI-augmented experience for a variety of design tasks.

There are a few reasons to take a data-first approach: First, it avoids the inefficient rebuilds that the moonshot approach tends to result in. It also acknowledges that ML projects are inherently for experiences with underlying statistical patterns and that those patterns are uncovered through data.

What does a data-first approach to machine learning look like?

It starts with collecting and labeling data so you have a minimum viable dataset (MVD) that informs a model. Read on for specifics on how to do both. Then you can build a minimum viable product (MVP) from that.

Using this method will help your team learn about the problem space, identify canonical edge cases, and even codify some basic rules about the problem. But, most importantly, you’ll exit the process with a valuable dataset that you can use to bootstrap your ML project.

First, ask: “What data do we already have?”

Perhaps you have raw data that just isn’t labeled. In that case, you’re in good shape. But what happens when you don’t already have data? How can you bootstrap your project to get it started and build toward a functioning machine learning system? That’s when it’s time to get creative.

Here, I’ll share how we tackled the two steps in our data-first approach..

Step 1: Collect Data

We’ve found three broad approaches for collecting data: Public (from available datasets), Harvested (from the environment), and Generated (from users).

Public Data

Public datasets form the basis for a ton of the ML projects today. For instance, Kaggle competitions are famous for providing datasets. In another famous example, many projects have used the Enron emails, released as part of the court battle in 2002. The sources are endless. A good place to start is by reading research papers related to your subject area to learn what data sources they are using.

In our project creating Einstein Designer, we found that papers often provide links to the datasets they used. This can be a great starting point as those datasets are typically high quality. For example we found the Rico dataset which has extensive data about mobile UIs.

Harvest Data

In other cases, the data you need might exist in the environment. That’s when you need to harvest it. Google Maps Streetview cars are a good example of this. All the location data exists in the real world. They just need to get it into a machine readable format.

In our case, we recognized that there is a ton of design data on the web, we just needed to build a crawler to go and collect it. This approach can generate a lot of really excellent data but you also have to contend with the grit and grime of the real world. Even when crawling the web we had to overcome many technical hurdles to find consistent ways of ingesting and labeling this data.

Generate Data

Lastly, there are times when the data needs to be created in the first place. In these cases, you’ll need to employ humans to get to work and help you out. For example, most websites and apps today collect a lot of user data. Every image you upload to social media adds to a giant body of images that those companies can use to train models. To make this approach work, you typically need a lot of active users.

We didn’t have access to tons of users we could employ to create data for us but we did use two related approaches to generating data: mechanical turks, and synthetic data.

Once you have some raw data, the next move is labeling it. This is both a challenge and an incredible opportunity to set a project up for success.

Step 2: Labeling Data

Machine learning isn’t magic. It requires labeling the data so that the models can encode the patterns within it. It’s important to have collected your data ahead of time before starting this process to ensure you’re labeling a consistent data set. There are many different ways of labeling data that can help lead to a successful ML model.

These are perhaps the three most common: Manual (with a mechanical turk), Heuristic (for raw datasets) and Synthetic (for created data).

Manual

The most obvious way to label data is to have humans do it by hand. This has the advantage of letting you leverage human intelligence. You can ask some pretty nuanced questions and still get back labels. While any one human might get things wrong, if you use enough people to label the data then you can either calculate the right answer. At the minimum, you can measure the subjectivity within your data. The team behind Bricolage used this technique to have human labelers specify what comparable components are across two different pages. Using this, they were able to discover that the labelers agreed ~78% of the time. The advantage of this is that it also established a benchmark to compare the model performance against. This is great because it can help measure and limit the inherent bias in the dataset. The downsides of this approach are the expense and varying data quality. Using platforms like Amazon Mechanical Turk, you can pay people to do your task or find colleagues to help out.

We did both — partly relying on in-house expert designers to label data for us. You’ll also need a tool that these labelers use to label the data. While there are off-the-shelf tools that can be very powerful like VGG, we also found value in building our own labeling tools optimized for specific tasks.

Heuristic

In some cases, you may be able to automatically label your data instead of hiring humans. This has the obvious advantage of being cheaper. The challenge here is to create a set of rules that can accurately label the data. If it was easy, then you probably wouldn’t need an ML model in the first place. So through this process, it’s recommended to focus on identifying true positives and true negatives. Since we don’t know what false positives and false negatives are, it’s better to filter ambiguous data out of the training set. Or at least label it as ambiguous to prevent introducing noise into your data.

For our project, we wanted to sort through thousands of web pages and label the parts that were product tiles. To do this, we created a set of rules that correctly identified only product tiles. Though it did miss a few along the way, this wasn’t important because we still collected a lot of good data.

Be careful though that your rules don’t accidentally introduce additional bias into your training data. One way to help prevent bias is to manually label a small but representative subset of examples. Using this manual data, you can write tests to verify that your rules work as expected. This is just test-driven development but used for the purposes of data gathering and labeling. When labeling product tiles, we took this approach and identified approximately 30 sites. For example: We found a site with a product image that was only 30px square. By writing a test for this case, we could see that our rules were too restrictive and then fixed this issue in our labeling code. In our case, we only labeled true positives but the tests enabled us to have greater confidence. Always follow up this process by inspecting the data to find new false positives and write more tests.

Synthetic

In the last approach, you both create the data and label it in one step. This has the advantage of being relatively easy to accomplish since you don’t have to deal with the messiness of the real world. The clear disadvantage is that synthetic data runs the risk of having the most bias since it wasn’t derived from real data. You’ll need to pay particular attention to crafting this data set to avoid bias or at least be aware of it.

To see how we created ours, check out my other article on creating synthetic data to learn more.

When your data is collected and labeled, it’s time to train your first iteration of the model and make your MVP.

Closing

Machine learning technology has the possibility of fundamentally changing how we do innovation. Maybe instead of the moon-shot metaphor, we could think of innovation using machine learning as being more like training a puppy. Unlike building a spaceship in which you might start by building half the systems, with a puppy you don’t start with half a puppy. Instead, you start with a small and slightly unruly version of a complete dog (model). Using this metaphor, your job is not to tightly engineer all of the puppies’ behaviors but to create experiences in which the puppy (model) can collect experiences (data) to learn to become a well-behaved dog. You bring the dog on walks to teach them about cars. You go to the dog park to learn about dogs. You visit friends to help them learn about people. And through it all you reinforce positive behaviors (labeling) so that the puppy will grow into a sociable, friendly, and obedient adult.

With machine learning we are training our innovations to perform tasks we desire of them. So we need to collect a wide array of unbiased data that has been labeled to reinforce the behaviors we want. And we need to continue to guide our models even after they’re built by maintaining and continuing to train them. Models, unlike traditional algorithms, are never done. There is always room to add more data, more labels, to increase the models performance and reliability.

In this version of innovation, it all starts with the data.

**Note: Prior to executing this approach, it’s a good idea to explore the product idea using cheap prototyping methods. Prototyping predictive systems can be a real challenge but using the traditional toolkit of rapid prototyping designers can test out various aspects of the experience and collect data about what products are likely to succeed in the marketplace. Then you’re ready to start.

Learn More

How to Generate Synthetic Data with Sketch

Thinking Like the Machine: Using Metrics to Generate and Quantify UX Design

The Role of Design in Machine Learning

Thanks

Sönke Rohde, Jessica Lundin, Michael Sollami, Alan Ross, Brian Lonsdorf, David Woodward

Learn more at www.salesforce.com/design

Follow us at @SalesforceUX.

Check out the Salesforce Lightning Design System

Machine Learning: Redrawing the Innovation Roadmap was originally published in Salesforce Designer on Medium, where people are continuing the conversation by highlighting and responding to this story.

Want to create some synthetic data of your own? This tutorial shows you how to create synthetic data to train a classifier that can answer that age-old question, “what is good design?” by predicting whether a design is good or bad. You can then extend these techniques to any UI component — for example, training a model to classify component types, or classify components by state. The possibilities are endless.

As an example, let’s generate a sample dataset of product tiles.

Setup

This tutorial relies on Sketch from Bohemian Coding; I recommend installing the latest version (≥63), along with two plugins that can make this process more efficient:

• Craft by Invision
• Rename It by Rodrigo Soares

Add Sketch Data

Now we need to collect the input data we’ll use to create our synthetic data. We’re creating product tiles, so we need product names, prices, and product images. To quote the Sketch documentation, “To create your own text Data source, create a plain text (.txt) file with each data value on a new line. For a new image Data source … create a folder with all the different images you want to use inside and add it via the Data tab in Preferences.”

To start, create a folder next to your Sketch file called Project Data. Create two plain-text files in TextEdit, named product_names.txt and product_prices.txt. Add ~25 entries in each, and save them in the Project Data folder.

To avoid introducing bias, make sure the values you create express a full range of data. For example. if all of your prices are under $100, or all in one currency, your model won’t perform well if it faces data that exceeds these limits.

Next, add 25 product images to a folder called product_images inside the Project Data folder.

To add these files as Sketch data sources, open Preferences and click the Data tab, then the Add Data button. Select the two text files and the folder. Click Done, and voilà!

Create “Good” Tiles

Down to business. In your Sketch file, create a variety of product tiles. Each one should use the same data, but vary in design, while still adhering to your definition of good design. If you have a well-defined brand standard, stick to its rules. Don’t worry, you’ll get an opportunity to break the rules soon.

We aren’t using nested symbols here, but I recommend them if you want to incorporate sublayouts, such as inline list price and original prices. Don’t use an artboard for each product tile — it’s easier if they are regular groups. As you build your tiles, be consistent in naming product image, product name, and price layers. Layer names become labels in the synthetic data you’ll use to train your model.

Let’s start by drawing a single tile. Remember to bind each text layer to Sketch data — when you add a text element for the name, use the Data menu to fill the layer with a product_name from your text file.

To create an image, draw a rectangle, then use the Data menu to select a product_image data source. This fills the background with an image, and crops it to fit the rectangle — making it easier to handle images of different sizes. Make sure to group your layers.

We want to use Sketch’s resizing and smart layout properties to ensure that each tile scales gracefully for every text value in your datasets. Use these properties with any nested symbols as well. Carefully consider edge cases — any bugs will reduce the accuracy of your model.

To create each new tile, duplicate this first tile — your layers will be named correctly and your data attached properly. Make as many tiles as you can; 50 is a solid start.

Now it’s time to organize and name your tiles. Select all tiles, select Align Left, then Tidy. Set vertical spacing to 50px. This should give you a neat single column of tiles. Reselect all tiles and select RenameIt, then enter Product%NNNN to give each of your tiles a unique name. Add enough Ns to ensure a unique name for each tile.

Create “Bad” Tiles

The data to train your classifier should have as many bad tiles as good ones. These are product tile designs that don’t meet your design standards. Got well-defined brand standards for fonts and colors? This is your opportunity to break the rules.

Start by selecting all the good tiles. Duplicate them into a new column below the original one, far enough away that the divide is clear.

With all the good tiles selected, click RenameIt and enter %*-Good. This appends -Good to your original names. Do the same for the bad tiles, this time with %*-Bad.

Now comes the fun part. For each bad tile, break the rules in a new way. Make the text overlap, or mess up the alignment. Get creative! As you do this, keep track of how you “break” each tile, and add that data to its name — for example, Product0001-Bad-misaligned.

As with the good tiles, do your best to support the range of data in your text files. If you overlap text, for example, make sure it will still overlap even if a shorter value is used. All the bad tiles must be bad!

Duplicate

Time for your hard work to pay off! Select all your tiles, then in the Craft plugin, select Duplicate. Since we’ve been organizing into a single column, use the horizontal checkbox and set the number to match your number or images and product names.

Next, delete the layers and groups that Craft added. Select the whole matrix of tiles you’ve made, and click Ungroup twice. Then delete the Duplicate Control layer.

Now select all the tiles again. In the Data menu, select Refresh Data.

Magic! You should now have a massive wall of 2500 unique product tiles.

Export Images

Ok, let’s save these tiles so you can start training models.

First, let’s use RenameIt to give each tile a unique name, so it saves properly. Select all tiles, click RenameIt, and enter %NNNNNN-%*. This prepends a unique ID to each tile, while preserving the template name and other labels.

In the properties panel, mark all tiles as exportable. In Sketch, you can choose export resolution; here I recommend 1x.

Click Export Selected and watch the data flow.

Later, when processing these images, you can parse the final file name, using “-” as a delimiter to get the unique ID, template ID, quality label, and reasons why a tile is bad.

JSON Data

In machine learning, images are great, but structured data is super useful. Thankfully, Sketch files are all JSON under the hood. The trick is to unzip the file. (If you’re using any symbols in your Sketch file, now is the time to detach them. The simplest way is to duplicate your file and delete the Symbols page to flatten the whole file.)

On a Mac, pop open your Terminal. Type unzip and a space. Drag your flattened Sketch file onto the terminal window, and press Enter. You should see the contents of your Sketch file in the same folder you’ve been working in.

Inside the new Pages folder, you should see a .json file with a long name, which contains all the JSON describing your tiles in a property called layers — including layer names showing which properties belong to which layers. (For more on file schemas, see the Sketch documentation.)

And that’s it. You’re now the proud owner of several thousand rows of synthetic training data. As you can see, it’s relatively easy to add more rows or more data — so grow your dataset to your heart’s content.

Happy modeling!

Thanks

Sönke Rohde, Jessica Lundin, Michael Sollami, Alan Ross, Brian Lonsdorf, David Woodward

Follow us at @SalesforceUX.

Want to work with us? Contact us at uxcareers@salesforce.com.

Check out the Salesforce Lightning Design System

How to Generate Synthetic Data with Sketch was originally published in Salesforce Designer on Medium, where people are continuing the conversation by highlighting and responding to this story.

When you’re working with machine learning, the traditional functions of design — crafting a product vision and communicating with stakeholders — apply, but ML also brings new factors to the table. This article explores how design techniques can be applied in ML development At base, it’s all about data — getting it (a lot of it!), cleaning it, understanding it, and ultimately building software on top of it. The process goes something like this:

1. Collecting data
2. Visualizing and cleaning that data
3. Creating models and algorithms
4. Evaluating those models and algorithms

Collecting Data

Design can contribute to data collection in several ways; the most direct is designing data collection tools. Think of this as a subset of product design, with a few key distinctions. Data collection tools are usually internal-facing, with the product team as the user. Productivity is key, and the tool needs to evolve along with the team. Data collection is also part of the discovery process. Labels and visualizations have a huge impact on team conversations and direction, and clear and effective UIs are crucial.

While in an ideal world, development teams have access to large, clean, public datasets, we often need to collect our own data. This part of the process is rarely published or discussed; most teams create their own solutions, which add time and cost. Sometimes you’ll have to log data over a period of time. Other times, the raw bytes exist but must be gathered into a unified dataset — making collection tools extremely valuable.

To figure out what features you need in a collection tool, view the problem through a product-design lens. For Einstein Designer, for example, we built a tool that reads a webpage, takes a screenshot, and uses the information in it to create a high-level preview of the site’s design system. We’ve run this process on many sites, and repeat it as we learn more about what information is important, so a tool that makes it more efficient was critical.

Some design contributions are more elemental. Take button and section labels, which often include critical terminology. Making them clear and consistent helps clarify design problems. In our data collection tool, we labeled one of the matrices and the list of individual text styles “palette,” which helped team members understand the scope of that term and differentiated it from other key concepts.

Data collected for machine learning is often unlabeled, lacking descriptors telling how to categorize it. For example, we might have a pile of images, but no descriptors about what they contain — e.g., cats or dogs. Often, humans must label the data.

To expedite this process for Einstein Designer, we built a game-like experience in which human users were challenged to classify product tile designs as good or bad, using keyboard arrow keys. Design quality was then defined by an image’s aggregate score. As designers, we contributed to this process by creating a a clear, functional UI and making the evaluation process as seamless as possible. The simpler the interaction, the more images users could evaluate. In this case, speed also forced users to make intuitive judgments. Since consumer users will also be making split-second decisions, this helped create analogous data.

Another exciting area of machine learning is synthetic data. Constructed to reflect expectations for real data, it’s used to bootstrap ML models before you have access to a large real dataset. For Einstein Designer, where we’re training models to understand design, we’re lucky to have Sketch, a powerful tool for generating new designs that can provide both an image of the generated design and a JSON representation that closely mirrors collected web data. We first train models on synthetic Sketch data, then later augment those same models with real data from the web.

We used Sketch to create 50 design templates, making heavy use of symbols and auto-layouts, so they’d still look good as the data changed — a process much like real web design engineering. We next used Sketch’s data features to wire up the templates with custom values. For each template we created good and bad versions, breaking the design in some specific way, such as overlapping or misaligned text. We then cloned these 25 times to generate 2500 labeled examples (50X2X25 = 2500). Each clone with attached Sketch data creates a unique example, resulting in a balanced dataset. Finally, we systematically labeled all layers.

Cleaning Data

The next step in ML development is cleaning your data. If you’re using synthetic data, this is already done — otherwise, some cleaning is necessary. Either way, visualizing the data will help you understand what’s actually in there.

Here again, designers have valuable skills to bring to the table. Whether you’re drawing possible plots on a whiteboard or creating them with Python or D3, you can use the design process to think through visualizations.

Creating Models

Once you have a clean data set and an understanding of its contents, the next step is to create algorithms or models based on your insights. While it can be tempting to jump straight into code and start training models, we’ve found it useful to first use lo-fi design prototypes to try out algorithms. For example, we prototyped the output of our generative design tool by simply creating a matrix of design options on a slide. We then asked our customers to eliminate any designs that didn’t meet their brand guidelines. This gave us confidence in the product requirements for our service and helped us understand what actions users might want to take.

You can also prototype the algorithm itself, to understand what kinds of predictions might be possible with given data. For example, we wanted to train a model to understand a brand’s existing style guide. We could collect individual styles used on a site, but wanted to understand how those styles work together. We realized we could prototype multiple approaches to this task and inform our intuition about how to use styles appropriately just by drawing them.

We created input data and wrote down the rules of the proposed algorithm, which team members then executed manually in Sketch. This process revealed that creating pairs of styles that follow each other in the original page was the best way to understand relationships between designs. It also gave us the confidence to start automating style relationship collection and further refine our methods.

This exercise also supported our intuition that certain algorithms wouldn’t work. Humans can’t perfectly simulate a machine learning algorithm, but machine learning isn’t magic; there must be some underlying pattern or information that tells the algorithm what to do. For example, we wanted to know how little information the design generator would need to pick an appropriate font size. We asked team members to draw designs based solely on data type and place in the design hierarchy, and quickly discovered that it’s necessary to know more about the values. Like a human designer, to make appropriate choices, the algorithm needs to know the approximate amount of text in each field.

Evaluating Models

Once you’ve created your algorithms and models, you need to identify, learn from, and correct mistakes and biases. Design can play an important role here, in creating tools to streamline the evaluation process.

For Einstein Designer, we needed to evaluate the output of our generative models. While there’s no easy way to automate scoring the quality of design, we could build a tool for evaluating the results. Our process drew on best practices from user research and design leaders including Charles Owens of Chicago’s IIT Institute of Design and Miles and Huberman’s Qualitative Data Analysis.

First we created a tool that would allow everyone on the team to visualize and review the same designs. It codified a set of test cases, created standardized inputs for each, and let us consistently test our system. Unlike in an actual product, where users should see only the highest quality output, it showed the model’s complete output to help us understand how it was functioning.

Next, each member of the team used the tool to independently rank designs, much as researchers might code a transcript to quantify qualitative data.

In Owen’s IIT Structured Planning course, for example, each student is asked to rank the relatedness of individual topics. The resulting data is then combined, and used to inform which themes are most central to a design project. Standardizing the questions students ask when comparing topics makes it possible to distribute work and cluster and compare the results. And having multiple students rank the same data helps reduce the impact of individual bias.

We took the same approach, having each team member rank the designs, then combining results to generate a more objective view of design quality, an inherently subjective topic. Repeating this process at the end of each design sprint helped us track our progress toward the goal of a robust design generation product.

As machine learning becomes ever more present in product design, remember to consider how design can play a role. The core design skills such as identifying and communicating needs, visualizing data, prototyping, building tools, and doing research all play important roles in the core machine learning process. Most of all, remember that the process is collaborative, everyone can participate, and more perspectives lead to better products.

Thanks

Sönke Rohde, Jessica Lundin, Michael Sollami, Tim Sheiner, Alan Ross, Brian Lonsdorf, David Woodward

Follow us at @SalesforceUX.

Want to work with us? Contact us at uxcareers@salesforce.com.

Check out the Salesforce Lightning Design System

The Role of Design in Machine Learning was originally published in Salesforce Designer on Medium, where people are continuing the conversation by highlighting and responding to this story.

UX design is no exception. Over the last year, Salesforce’s Einstein Designer team has been working to create models and algorithms that generate design variations, which human designers can adapt or riff on, speeding the design process and helping designers stretch their creative muscles. Our vision is to create multiple machine-learning models that can help people and companies create better designs, faster. Even professional designers can benefit from such tools, which accelerate the design process and free up designers to focus on user needs and higher-level problem solving, improving the user experience.

Impressive goals. But first we needed to build the models — and to do that, we had to think like the machine.

The Challenge of Quantifying Design

When taking on a machine learning problem, one of the first questions to ask is, “What data do we have?” And the kind of data we needed — codified design rules presented in ways the machine can understand — wasn’t exactly hanging out in a large, publicly available database, ready to hop into our models. We had to create it ourselves. We thought more deeply about how to generate data about design that we could use to train models, and soon realized that what we needed were quantitative ways to measure design.

OK, measuring a creative process that blends the intuitive and the practical. No problem, right?

One natural place to look, we thought, would be the design curriculum, where foundational principles such as Gestalt theory could help us measure aspects of design. In practice, however, we found these principles to be too fine-grained, and too tied to human perception. It’s easy for a human to identify a figure on a ground, for example, but it turns out to be ridiculously hard to create a computational model of that phenomenon. Just look at the decades of work in computer vision it’s taken to get computers to identify objects in photos with some level of accuracy.

What we needed instead, we realized, were metrics that are measurable in the design itself.

A Search for Metrics

So who’s successfully measuring aspects of UX design? That question led us to the accessibility community, which has developed a robust set of metrics for evaluating design standards. For example, color contrast is a well-defined metric that measures and assesses the readability of any given piece of text. But while accessibility metrics are fantastic starting points, and must be integrated into any generative design system, they aren’t sufficient to fully measure a design.

Design is fundamentally a method for organizing information for communication. So to quantify a design, we need metrics that capture how its information is organized. With this idea as our guide, the Einstein Design team identified three major aspects of design that can be measured and used to express that design’s intent: grouping, prominence, and coding.

Grouping is the practice of sorting aspects of a design into groups based on their relationships. For example, we might group a title and body text, or an image and its caption. Together these two groups combine to form a larger grouping — an article tile.

Prominence describes perceptual relationships between elements and identifies a hierarchy of perception, in which the controlling element is perceived first. Prominences can be represented in a tree-like information hierarchy. For example, a title should ideally be seen before its supporting paragraph, which in turn is seen before the associated image caption.

Coding is the use of symbolic references to associate an element with a class of elements. Coding creates meaning without using words. For example, a blue rectangle with rounded corners and white text tells users that a design element is a button. In early UX design, skeuomorphic elements taught users what they meant using the design language of the analog world.

Measuring Metrics

Ok, now that we’ve identified these three important metrics, how do we measure them?

Grouping

With grouping, our measurement work was partially done — HTML is fundamentally a tree structure that describes the grouping of elements. The problem here: Data collected from the web is noisy. There are hundreds of ways to build a website, and web design techniques are becoming ever more complex and diverse. And while it would be awesome if every web page featured perfect semantic markup, the reality is more sobering. So on top of the HTML information for each web page we analyzed, we analyzed pixel distance, element alignment, and a handful of other heuristics. With this data set, we built an initial grouping metric that we used to bootstrap our data collection process.

When measuring any design metric, you also need a way to quantify and analyze your accuracy, so you know how to improve your metric. We validated the direction by having human testers identify related text elements. The team behind Bricolage used a similar method, asking multiple humans to manually label data, then comparing agreement between individuals to create a baseline accuracy for the metric. The benchmark for the automated solution is set by human testers who agreed on ~70% of a design’s labels. In essence, this is a way to separate objective and subjective aspects of design.

Prominence

Like grouping, prominence is defined by a combination of subjective and objective qualities. We started by evaluating text prominence, developing a scoring method designed to sort text styles from most to least prominent. Using font size, font weight, and text/background contrast, we created a useful approximation; while individual text comparisons may be debatable, most people would find the text at the top of the list more prominent than that at the bottom. Most important, a rudimentary metric like this one can be used to bootstrap more sophisticated models. Here again, we created tools that let human testers compare the prominence of two pieces of text, and used the results to generate a baseline accuracy metric.

This process revealed one immediate limitation: Prominence is highly contextual. Placed side by side, a bold font is more visible than a thin font. But a thin font that is large enough, with enough space around it, can be more eye-catching than the bold text. Color is another factor. Generally red is more eye-catching than other colors, but on a site where red is used frequently, a complementary green will stand out. We can use computer vision models to estimate visual salience and account for many such contextual effects.

Coding

With coding, the play of subjectivity and objectivity is driven by viewer experience. For example, with the transition from skeuomorphic to flat design, users have become accustomed to a new UI language. Coding is highly dependent on the design zeitgeist: For someone to think that something looks like a button, they first need to have experienced similar buttons. This raises the possibility that, as in fast fashion, visual designers could plot the evolution of button design over the decades, then project forward to predict the next big style.

In one of our first attempts at measuring coding, we took screenshots of all Salesforce Lightning Design System (SLDS) components, clustered them based on visual appearance, and used our knowledge of what each one is and does to generate a rich labeled dataset. We can use this to create models that generate designs for new components, that look like other components with similar functionality. For example, when generating designs with buttons, a model could ensure that the new buttons appear similar to those in the SLDS system, without explicitly limiting the generator to their templates.

A foundation for quantifying design success

These three metrics — grouping, prominence, and coding — lay a foundation for quantifying design, and eventually for building machines that know how to design for humans. The resulting models crystalize the objective aspects of design while leaving room for designers’, and users’ subjective perception. Building on concepts at the root of accessibility, they offer measurable metrics and tools that people and companies can use to create humanist, human-friendly designs that communicate their function clearly and consistently.

Thanks

Sönke Rohde, Jessica Lundin, Michael Sollami, Tim Sheiner, Alan Ross, Brian Lonsdorf, David Woodward

Follow us at @SalesforceUX.

Want to work with us? Contact us at uxcareers@salesforce.com.

Check out the Salesforce Lightning Design System

Thinking Like the Machine: Using Metrics to Generate and Quantify UX Design was originally published in Salesforce Designer on Medium, where people are continuing the conversation by highlighting and responding to this story.