Einstein Designer — AI-Powered, Personalized Design at Scale

Sönke Rohde
Jun 23 · 10 min read

Since the dawn of manufacturing, there’s been a hard limitation on how we develop products. This applies to traditional manufacturing as well as to software development, which is the focus here. The conventional wisdom is that either quality, time-to-market, or cost has to be compromised. Or as the saying goes: “Good, Fast, Cheap. Pick two.”

Image for post
Image for post

This truism is rooted in a pre-AI (Artificial Intelligence) world. The power of AI changes the rules of the game.

For example, at Salesforce Customer Success is one of our most important values. That implies that we will not compromise on quality or time-to-market. The challenge is: how can we make it efficient (cheap) to deliver high-quality (good) products at an industry-leading pace (fast)?
One of the most time consuming and expensive aspects of a software project is User Interface (UI) development. While today we have programmatic and declarative solutions for developing UIs, both approaches fall short when it comes to truly changing the rules of the game. It still takes significant time and substantial resources to build good user interfaces. UI development still eats up a significant portion of the product development lifecycle. What would be the fastest possible way to create UI? Not doing it manually at all. But what does that even mean?

The Future of UI

To research this challenge and discover if AI can be trained with aesthetic values, we founded the UX R&D team at Salesforce. Our initial vision statement:

We radically improve the way to create user interfaces by applying Artificial Intelligence and Machine Learning.

Data Science for Design

To develop an intuition for the space and the potential inputs for a model, we kicked off a project called Deep Learning UX, or DLUX for short (say “deluxe”). As our initial training data we picked the Fortune 500 and unicorn companies (start-ups with valuations over $1 billion) with the hypothesis that these sites have been created by professional design agencies and meet some level of quality (we are fully aware that this training data is potentially biased). We analyzed roughly 1,000 sites, focusing on their use of color and typography. We were lucky to collaborate closely with the data visualization specialist Moritz Stefaner who has been producing Truth & Beauty, literally.

Color

Sliced Color Palettes
Sliced Color Palettes
Data Viz by Moritz Stefaner

Another very interesting lens on color is the overall distribution. In the visualization below, each dot represents the occurrence of color in a palette across the 1,000 sites analyzed.

  • Size = amount
  • Angle = color hue
  • Distance to center = +-saturation sat (+ lightness correction)
Color Distribution
Color Distribution
Data Viz by Moritz Stefaner

The major takeaway is that there is a strong blue-red axis. Purple and green are being used much less. This is only one lens into our training data and not representative for the entire internet.

Another way to slice this data is by industry which reveals the unsurprising fact that software companies gravitate towards blue.

Color distribution by industry
Color distribution by industry
Data Viz by Moritz Stefaner

Typography

Font Size Distribution

Image for post
Image for post
Data Viz by Moritz Stefaner

Our next question was: are there patterns for how font sizes are applied to the elements of a webpage? To answer this question we tried to label the data with HTML tag names. HTML tags are intended to describe semantic meaning but the training data didn’t yield any meaningful correlations as very little production HTML is semantically correct. The web is messy and SEO is likely to play a role too.

Typographic Hierarchy

A key aspect of good typography is the creation of hierarchy to establish an order of importance within a design. We’ve analyzed pairs of typographic styles to develop an intuition on how we might encode type hierarchy in a scalable way. Below are examples for an arc and radial arc diagram illustrating how styles are used together on one particular site. Each character represents a distinct font style for an entity in our training data. The weight of the line expresses the strength of the relationship between the styles.

Typographic relationships
Typographic relationships
Data Viz by Moritz Stefaner

Latent Style Space

Personalized Design

“Cheap Changes Everything”

This quote from the book Prediction Machines gets to the essence of how transformational AI can be. Once something becomes cheap, it can change everything. If UI creation is suddenly cheap, what would it enable?

One of the most exciting applications we are currently working on is the personalization of design. If it was cheap to generate UIs it would be possible to create multiple versions of a UI. Imagine a custom UI for every person who visits a website. Wait — what?

What does it even mean to personalize design?

Let’s take a look at a scenario in the commerce space. Meet our two shoppers Sheryl and Shawna. While shopping on a site, they both see relevant content (already predicted through AI) but the presentation/design isn’t really resonating with either of them. Sheryl is a bargain hunter, she likes to look for discount prices. While the discount appears in the default design, it’s not very prominent. Shawna usually makes her buying decisions based on product ratings which are entirely missing in this design. To satisfy both their personal preferences, we would need to highlight the information they personally care about: Sheryl would get a prominent treatment of the discount price, and Shawna can see the product ratings front and center.

Personalized Design Scenario
Personalized Design Scenario
Design by Owen Schoppe

That’s personalized design, because one size does not fit all. Everybody is unique and we all care about different aspects of a product and we all apply our own personal (aesthetic) value judgment.

Personalized Design
Personalized Design
Design by Owen Schoppe

Scale is Key

  1. The creation of design variations needs to be automated. Creating them manually does not scale (it’s too expensive and time-consuming). Generative design is the answer.
  2. The personalization, or who sees what design variation, needs to be automated. Mapping design variations to audiences manually has the potential of being flawed, can be a cumbersome process, and can’t scale to millions or billions of users. It needs to be a prediction.

Generative Design w/ Deep Learning UX

Augmented Intelligence

Artificial Yet Ethical

“The ethical and humane use of technology in the Fourth Industrial Revolution is the way forward — not just for our industry but for humanity. We have to make sure that technology strengthens our societies instead of weakening them. Technology needs to improve the human condition, not undermine it.”

We see DLUX as augmented intelligence, amplifying the creative process, and enabling Salesforce customers to deliver innovative, bespoke experiences. We are not ignorant of the potential for bad actors when it comes to this type of technology, just the opposite: it’s part of our process to ensure the AI technologies we create are aligned with our values.

For example, we conduct workshops to scan for possible negative ramifications of our technology as a part of our product development lifecycle. UK based think-tank DotEveryone developed Consequence Scanning Workshops to guide organizations in creating technology responsibly. We apply their framework at the start and during the process, instead of attempting to ethically retrofit our technology after the fact.

For more information please see Salesforce’s Ethical and Humane Use site.

Predictive Personalization w/ Bandits

“In probability theory, the multi-armed bandit problem (sometimes called the K-[1] or N-armed bandit problem[2]) is a problem in which a fixed limited set of resources must be allocated between competing (alternative) choices in a way that maximizes their expected gain, when each choice’s properties are only partially known at the time of allocation, and may become better understood as time passes or by allocating resources to the choice.”

Einstein Designer

Einstein Designer Overview
Einstein Designer Overview
Design by Sönke Rohde

What started as a moonshot idea, generating design with AI, has enabled us to take personalization to a whole new level.

Einstein Designer was first introduced at Dreamforce ’19. Please check out the video here (1:09:50–1:12:50).

UX Design Award 2020

Image for post
Image for post
UX Design Awards 2020 Nomination

Fast Company — Innovation by Design 2020

Image for post
Image for post
Fast Company Innovation by Design 2020 Honoree

What’s next?

Thanks

Salesforce Experience and Design

A collection of stories, case studies, and ideas from…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store