Einstein Designer — AI-Powered, Personalized Design at Scale
Since the dawn of manufacturing, there’s been a hard limitation on how we develop products. This applies to traditional manufacturing as well as to software development, which is the focus here. The conventional wisdom is that either quality, time-to-market, or cost has to be compromised. Or as the saying goes: “Good, Fast, Cheap. Pick two.”
This truism is rooted in a pre-AI (Artificial Intelligence) world. The power of AI changes the rules of the game.
For example, at Salesforce Customer Success is one of our most important values. That implies that we will not compromise on quality or time-to-market. The challenge is: how can we make it efficient (cheap) to deliver high-quality (good) products at an industry-leading pace (fast)?
One of the most time consuming and expensive aspects of a software project is User Interface (UI) development. While today we have programmatic and declarative solutions for developing UIs, both approaches fall short when it comes to truly changing the rules of the game. It still takes significant time and substantial resources to build good user interfaces. UI development still eats up a significant portion of the product development lifecycle. What would be the fastest possible way to create UI? Not doing it manually at all. But what does that even mean?
The Future of UI
What if we could generate user interfaces automatically? To take it even a step further, what if we could predict UIs?
To research this challenge and discover if AI can be trained with aesthetic values, we founded the UX R&D team at Salesforce. Our initial vision statement:
We radically improve the way to create user interfaces by applying Artificial Intelligence and Machine Learning.
Data Science for Design
In order to train a machine-learned model, we needed to understand the space we are working with. One of the first questions a data scientist will usually ask when it comes to training a model is: “What are the inputs and what kind of outputs are you looking for?”
To develop an intuition for the space and the potential inputs for a model, we kicked off a project called Deep Learning UX, or DLUX for short (say “deluxe”). As our initial training data we picked the Fortune 500 and unicorn companies (start-ups with valuations over $1 billion) with the hypothesis that these sites have been created by professional design agencies and meet some level of quality (we are fully aware that this training data is potentially biased). We analyzed roughly 1,000 sites, focusing on their use of color and typography. We were lucky to collaborate closely with the data visualization specialist Moritz Stefaner who has been producing Truth & Beauty, literally.
Color
By analyzing the training data, we quickly came to the conclusion that there isn’t a single color palette for a site or page, but that web design today favors slices — using different color palettes for different portions of a page/site. For instance, we may slice a page into the salient components, such as a header, a call-to-action (CTA), content blocks with alternating backgrounds, and different kinds of footers. The data visualization below shows the corresponding color palette for each slice, with the size of each color block indicating the proportional usage. Further analysis also led to algorithms that revealed the semantic meanings of how the colors are being used. To use AI terminology: we found ways to label the data.
Another very interesting lens on color is the overall distribution. In the visualization below, each dot represents the occurrence of color in a palette across the 1,000 sites analyzed.
- Size = amount
- Angle = color hue
- Distance to center = +-saturation sat (+ lightness correction)
The major takeaway is that there is a strong blue-red axis. Purple and green are being used much less. This is only one lens into our training data and not representative for the entire internet.
Another way to slice this data is by industry which reveals the unsurprising fact that software companies gravitate towards blue.
Typography
We looked at typography by analyzing font sizes, which revealed a very interesting insight: the data shows a strong preference for even font-sizes which is unlikely for aesthetic reasons but to make it easier to calculate than odd numbers. It’s basically evidence that the training data has been designed/developed by humans.
Font Size Distribution
Our next question was: are there patterns for how font sizes are applied to the elements of a webpage? To answer this question we tried to label the data with HTML tag names. HTML tags are intended to describe semantic meaning but the training data didn’t yield any meaningful correlations as very little production HTML is semantically correct. The web is messy and SEO is likely to play a role too.
Typographic Hierarchy
A key aspect of good typography is the creation of hierarchy to establish an order of importance within a design. We’ve analyzed pairs of typographic styles to develop an intuition on how we might encode type hierarchy in a scalable way. Below are examples for an arc and radial arc diagram illustrating how styles are used together on one particular site. Each character represents a distinct font style for an entity in our training data. The weight of the line expresses the strength of the relationship between the styles.
Latent Style Space
The key takeaway from our initial data science is that there is a meaningful latent space when it comes to style encapsulating aspects around color, typography, and more. The latent space is basically a representation of compressed data which in our use-case enables the featurization of design rules and heuristics. It’s our foundation for what we call Deep Learning UX to produce generative design. But how is this useful?
Personalized Design
“Cheap Changes Everything”
This quote from the book Prediction Machines gets to the essence of how transformational AI can be. Once something becomes cheap, it can change everything. If UI creation is suddenly cheap, what would it enable?
One of the most exciting applications we are currently working on is the personalization of design. If it was cheap to generate UIs it would be possible to create multiple versions of a UI. Imagine a custom UI for every person who visits a website. Wait — what?
What does it even mean to personalize design?
Let’s take a look at a scenario in the commerce space. Meet our two shoppers Sheryl and Shawna. While shopping on a site, they both see relevant content (already predicted through AI) but the presentation/design isn’t really resonating with either of them. Sheryl is a bargain hunter, she likes to look for discount prices. While the discount appears in the default design, it’s not very prominent. Shawna usually makes her buying decisions based on product ratings which are entirely missing in this design. To satisfy both their personal preferences, we would need to highlight the information they personally care about: Sheryl would get a prominent treatment of the discount price, and Shawna can see the product ratings front and center.
That’s personalized design, because one size does not fit all. Everybody is unique and we all care about different aspects of a product and we all apply our own personal (aesthetic) value judgment.
Scale is Key
In order to create personalized design at scale, two processes need to be automated:
- The creation of design variations needs to be automated. Creating them manually does not scale (it’s too expensive and time-consuming). Generative design is the answer.
- The personalization, or who sees what design variation, needs to be automated. Mapping design variations to audiences manually has the potential of being flawed, can be a cumbersome process, and can’t scale to millions or billions of users. It needs to be a prediction.
Generative Design w/ Deep Learning UX
Deep Learning UX (DLUX) enables us to generate design at scale. DLUX automatically learns the design system of an existing brand by analyzing its website and has been trained to produce AI-generated, brand-aligned design variations.
Augmented Intelligence
Humans choose which generated design variations — Augmented Intelligence or Human-In-The-Loop in AI parlance. DLUX amplifies the creative process by generating a universe of designs that can be clustered and ranked based on different criteria. Then the human-in-the-loop calibrates the design generations and ultimately selects the design variations. This process gives users visibility into how the AI works and control over how their business is presented. We believe this process helps foster a trusted relationship between our customers and the application of generative design.
Artificial Yet Ethical
Developing new technology, especially in the AI space, requires a responsible approach. Salesforce’s chairman and CEO Marc Benioff summarizes it best:
“The ethical and humane use of technology in the Fourth Industrial Revolution is the way forward — not just for our industry but for humanity. We have to make sure that technology strengthens our societies instead of weakening them. Technology needs to improve the human condition, not undermine it.”
We see DLUX as augmented intelligence, amplifying the creative process, and enabling Salesforce customers to deliver innovative, bespoke experiences. We are not ignorant of the potential for bad actors when it comes to this type of technology, just the opposite: it’s part of our process to ensure the AI technologies we create are aligned with our values.
For example, we conduct workshops to scan for possible negative ramifications of our technology as a part of our product development lifecycle. UK based think-tank DotEveryone developed Consequence Scanning Workshops to guide organizations in creating technology responsibly. We apply their framework at the start and during the process, instead of attempting to ethically retrofit our technology after the fact.
For more information please see Salesforce’s Ethical and Humane Use site.
Predictive Personalization w/ Bandits
Once we have a set of good design variations one task remains: Who is going to see which design variation? As mentioned above doing the segmentation manually is an error-prone and time-consuming endeavor. In contrast to classical A/B or multivariate testing, the goal, in this case, is not to find the one design that works OK for everyone, but to find the best design for each individual. The industry-leading approach for this scenario is a multi-armed or contextual bandit. Quoting Wikipedia:
“In probability theory, the multi-armed bandit problem (sometimes called the K-[1] or N-armed bandit problem[2]) is a problem in which a fixed limited set of resources must be allocated between competing (alternative) choices in a way that maximizes their expected gain, when each choice’s properties are only partially known at the time of allocation, and may become better understood as time passes or by allocating resources to the choice.”
Einstein Designer
Everything we’ve discussed above is becoming a reality in Einstein Designer. No code, no templates, Einstein Designer is an industry-first AI-powered generative design solution that automatically delivers personalized designs that delight every customer. Einstein Designer combines generative design using DLUX and predictive personalization to enable personalized design at scale.
What started as a moonshot idea, generating design with AI, has enabled us to take personalization to a whole new level.
Einstein Designer was first introduced at Dreamforce ’19. Please check out the video here (1:09:50–1:12:50).
UX Design Award 2020
Einstein Designer — Powered by Deep Learning UX won a nomination for the UX Design Awards 2020 #UXDA20.
Fast Company — Innovation by Design 2020
Einstein Designer — Powered by Deep Learning UX is a finalist in the Data Design category and an honorable mention in the General Excellence category in Fast Company’s 2020 Innovation by Design Awards. Read more here. #FCDesignAwards
What’s next?
- We are actively working on publishing the interactive versions of the beautiful data visualizations and more exciting research insights around DLUX. Please stay tuned.
- Follow me on Twitter to get more updates on DLUX and Einstein Designer.
- Fast Company: The future of user experience design starts with AI
- Owen Schoppe: Thinking Like the Machine: Using Metrics to Generate and Quantify UX Design
- Owen Schoppe: The Role of Design in Machine Learning
- Owen Schoppe: How to Generate Synthetic Data with Sketch
- Follow us at @SalesforceUX
- Want to work with us? Connect uxcareers@salesforce.com
- Check out the Salesforce Lightning Design System
Thanks
Alan Ross, Brian Lonsdorf, David Woodward, Jessica Lundin, Kathy Baxter, Madeline Davis, Michael Sollami, Moritz Stefaner, Noah Guyot, Noelle Moreno, Owen Schoppe
