The UI of AI
Note: This is a rewrite of a post originally published in Czech language.
More than one month ago, I had an opportunity to give presentation at a local meetup UX Reality, which took place in the MSD IT Innovation Center in Prague.
Meetup organizers asked me to talk about the relationship of UX and AI, so I decided to focus on the very interface between humans and AIs, in particular those AIs, which are powering some of the most popular today services — Facebook, Google or Amazon.
How does the User Experience of these services work? What is the relationship between humans and AIs? How the artificial machine becomes intelligent? How does it learn? And if both sides of the human-machine interface become intelligent, shouldn’t we start to talk, in addition to the “User Experience”, also about the “Machine experience”?
What is the Intelligence?
The “Internet definition” says:
This definition comes to me as too strict and narrow. I like more the definition of former Supreme U.S. Court’ Judge Potter Stewart from 1964:
I shall not attempt to define it, but I know it, when I see it.
It is worth to mention, that judge Stewart in this case didn’t speak about intelligence, but pornography. However, both intelligence and pornography are subjective phenomena with unclear parameters. And although most people would have troubles coming up with a clear definition for either of those, they most likely recognize both of them when they see them.
A bit of history
Artificial Intelligence is a subfield of Computer Science with rich history, reaching back to at least the first half of the 20th century. However, for the purpose of studying the relationship of AI and UX, we can start much later.
80s: Experts are modelling the world
In the 80s, there was a boom of the so-called “expert systems”, which attempted to model the world by using rules. Simply put, programmers and business analysts were convinced, that they can analyze everything and express the world using a finite number of “if-then-else” rules.
This approach (not much different from the socialist attempts for central planning) has quickly reached its limits, because the world is too complicated, constantly changing and unpredictable.
So as a result, in 1984, there came a crash, followed by a period which became known as “AI Winter”. During this era, the whole field of AI pretty much froze for more than 20years.
90s: Spam filters, which learn
Towards the end of 90s, then popular email services started to leverage new technique for recognizing and filtering spam. It was based on statistical methods and had an ability to actually learn to recognize spam emails. These systems would first have to be trained on a large set of messages, which were previously marked by users as spam. Result of this training process would be a statistical model able to decide with quite high degree of certainty, whether a new incoming message is a spam or not.
These spam filters (as opposed to the expert systems in 80s) were not trying to describe email spam with rules, but they literally learned to recognize spam based on past experience, represented by the training set of messages previously annotated by human users.
This is a very important concept — using the wisdom of the crowd (e.g. a lot of users annotating spam messages) to teach the machine.
00s: Web2.0=Read/Write Web
Few years after the Dot-com, new types of services have emerged, allowing common users to publish and share any multimedia content, like links on del.icio.us, blog posts on blogger.com, photos on flickr.com or videos on youtube.com. These services have effectively given common users the write permissions to the Web as a publishing platform.
First social networks have emerged almost in parallel to that. For the majority of internet users, they quickly became primary sites for accessing, sharing, liking and commenting content. Social networks have further amplified an explosion of user-generated content on the internet.
The following infographics from 2016 shows how much content was created and consumed using popular internet services in one average minute(!!!):
2010: Model the mind, not the world
If social network services wanted to stay being useful for their users, they had to employ new techniques for searching and filtering through these growing volumes of user-generated data.
Around the year 2010 started a renaissance of AI in which (as opposed to 80s), we don’t model the world, but the mind, which can learn about the world.
The flagship concept of this era are various types of so-called deep neural networks, which are inspired by the structure of the human brain. They are composed of interconnected layers of neurons, where each layer works on a higher-degree of abstraction than the one below.
The learning process is similar to the one used for spam filters. The neural network is first exposed to a training dataset which effectively results in a change of weights of individual neurons and their connections.
Learning from signals
Regardless of its concrete internal architecture, we can look on machine learning as on learning from various external signals. In the case of image recognition, signals are individual pixels, in the case of spam filters individual email messages.
For the AIs powering big services like Facebook or Google, signals are both the content created by users (text, images, videos) as well as all their activities, which they perform in regards to that content as well as to each other — what they create, read, share, like, comment, who they send messages to, etc.
The machine is using us
In 2007, the video below, called “The machine is using us”, was released:
Although it is more than 10 years old, its main message is even more relevant today than it was back then:
Everything we do online serves as an input for machine learning.
Example: How Facebook AI is using us
Let’s use Facebook as an example of a system, which is literally using us (people) for its own learning.
Facebook’s main business model is advertising — Facebook is a platform for showing ads to its users. When the user clicks on the ad, Facebook gets paid by the advertiser who has put that ad into its system.
Facebook’s most important business metrics thus are:
- the total amount of time users spend on the site or within the app
- how often they return.
- the % ratio between content clicked/content shown
The more often users visit Facebook and the more time they spend on it, the higher the chance they’ll click on some content, ideally an ad.
Facebook AI is thus optimized to motivate its users to come back often and to stay long, by showing them content which is constantly changing and which is the most relevant for them. This content is created by other Facebook users and its relevancy for the given user is determined by users rich profile, composed of everything Facebook knows about him — all his activities, all the content he has created, his mutual affinity to other users, etc.
Each user is thus provided with his own view on all the content and activities which the Facebook consists of. This tailoring of service and its content is called personalization and it is one of the most important end-user applications of Machine Learning in todays digital products and services.
Dopamine for Data
In the core of Facebook personalization, there is a very important feedback loop between Facebook (the machine) and its user (the human) which holds the whole service together:
- User provides his data — personal data, list of friends and most importantly his activities and interactions
- On each visit, Facebook provides a different content, personalized to the user’s profile.
- The user is happy. Literally. User’s happiness is caused by a neurotransmitter called Dopamine, which is released by his body as a reaction to the variability of the provided content.
This concept is well known in the traditional TV programming and is directly tied to metric “jolts per minute”, which equals to how many times the given scene on the screen changes.
This is why Facebook gives you a different version of News Feed every time you refresh your browser or sends you notification every time someone likes or comments on your photo. This is also why passive browsing through your Facebook feed feels similar to endless channel switching on your TV remote.
Facebook’s AI works on the principle “Dopamine for Data”. The more we use the service and the more data we give it, the more dopamine kicks we get in return.
This basically caused most people to become addicted to Facebook. And not just Facebook — a plenty of other today digital services and applications work on the same principle. This is why in 2012, Mark Zuckerberg announced smartphones to be a primary future platform. This is also why in the same year, Facebook bought Instagram, another service with a high dopamine distribution potential.
How to design similar addictive products and services is described in the book Hooked: How to build habit-forming products:
Giant Quantitative User Research
A required prerequisite of every good User Experience is User Research. User Research is a tool used by designers to get better understanding of users of their product, of how they think and what are their needs. There are essentially 2 types of User Research – quantitative and qualitative.
Quantitative research is based on statististically processing large amounts of discrete data, collected through digital platforms for capturing users digital trails, like Google Analytics. The collected datasets essentially describe how users behave within the context of a given product or service. They allow to answer the question „What users do“ with the service – what they click on, in what order, how long they stay engaged, etc.
The AIs of Facebook and other digital services are continuously collecting similar quantitative data about us. These AIs are in effect running a continual large-scale user research on us. How large this research is shows the following picture:
The above numbers do not show the whole picture though, because we are giving our data to Facebook and Google not just when we physically visit their site, but also other websites or mobile apps.
Every website which uses Google Analytics for collecting data about its users, serves as as a source of data for Google itself too. Similarly Facebook collects data from every website, which contains the Like button on it or which allows its users to login using their Facebook credentials.
The volume and richness of user data available to Facebook and Google AIs for learning is thus much larger and collected from all around the web.
Study and learn to become someone!
That’s what they were telling me when I was a kid. Similar to that, if AIs of Facebook, Google and a likes are learning from us, who will they become?
Let’s play a little game. Write down the first identity, personality or job which first comes to your mind, when you hear: Facebook. Google. Amazon. Tesla.
Got it? How did it go? For me, it ended up as follows:
Facebook is a newspaper seller kid, standing on the corner and shouting on everyone passing by. Google is Dr.Know, who knows answer to every question. Amazon is a professional seller, who knows everything about transactions – how and when they are likely to happen and why. And Tesla is an ambitious driver, very precise and very fast.
Similar personification of AI through its learning is not just a mental excercise but reality.
For example last year, Microsoft deployed twitter bot named „Tay“. Tay was supposed to be a teenage girl with the abilities to lead simple conversations with other twitter users and to learn from those conversations. However, after 24hours, Microsoft had to turn Tay of, because people (trolls) have literally turned her into a Hitler-loving sex machine. Tray was essentially like a good student, reading wrong books.
From shared AI to personal AI
When we use different apps like Microsoft Office or Adobe Photoshop, they are just tools for us.
When we use web services like Google, Facebook or Amazon, we percieve them as sort of virtual institutions usually without associating them with human-like attributes.
On the other hand new class of apps like Siri, Cortana, Alexa and others represent personal assistants, with their own unique personalities.
One of the first cases of widely deployed virtual asistant with its own personality was ingenious Mr Clippy brought into Microsoft Office in 90s. While he was originally supposed to be helping the MS Office users, he ended up bothering them. Why? Because he was designed to proactively suggest things and actions, while running on PCs with a very limited computing power, which in turned limited his intelligence.
Poor Mr Clippy ended up as a proactive dumb begging for attention and hated by pretty much everyone.
Voice is the organ of the soul
Creators of today smart asistants „living“ in our smartphones and smart speakers seem to have learned from Mr Clippys mistakes. They are reactive, waiting for our command or question. We ask them with our voice and they have a potential for limitless intelligence, because they are backed by the cloud computing infrastructure.
For now, they only know answers to simple questions and they can acomplish simple tasks („What is the weather today?“, „Call me an Uber“) but that is just a first step. As will their intelligence grow, they will be able to lead longer conversations with us, understand the context and remember topics we spoke about last time.
How will our relationship with machines change, when they will actually start to understand us?
Imagine the following situation from common life – Man is coming back home.
„Hi, I am home“.
„Hi, how was your day? Did you make it for the release?“
„Yes, we did, but it was pretty close. Josef was finishing the memo till the last minute“.
„I would not expect anything else. Last time it was the same, wasn’t it? Do you want something for dinner?“
„I’d go with salad“.
„Chicken or cheese?“.
„Probably chicken“.
„It will be ready in 10 minutes“.
„Is the new episode of Game of Thrones available?“
„Yes, we can watch it while you’ll eat”.
Nothing special, right? Just except the guy is single, lives alone and is not a schizophrenic. He is having such dialogs with his personal AI, which has its own style of speech, which knows his favorite kind of salad and which remembers conversation topics from the past, including important facts, people and events.
Qualitative User Research
Apart from the quantitative research, which can provide insights into “What happened” and which we’ve discussed above, there is also the qualitative part, which leads to understanding “Why something happened”. Qualitative research methods (like interviews or usability studies) are based on observing and talking with users.
Let’s demonstrate the relationship between the quantitative and qualitative research on the example of an ecommerce site owner who in his Google Analytics sees, that 90% of people which added some product into the shopping cart did not finish the purchase and left the site without ever coming back. The site owner knows WHAT is happening. By testing the shopping cart with real users he finds out, that they are having hard time finding the “Next” button, because it has insignificant color and is placed in the unexpected part of the screen. The site owner now knows WHY users dont finish the purchase and can also do something about it.
As we’ve discussed above, AI is basically running a continuous quantitative research based on discrete activities, which we (the people) perform using interfaces composed of clickable components. This communication channel between people and AI is thus determined in the design time of the application, when the whole interface is refined by the designer and engineers.
This communication channel will get much broader and richer, once people start to have regular conversations with the AI. The machine will do an instant sentiment analysis of our voice, it will immediately react to its changes, it will be able to ask additional questions. It will most likely learn more from 5 minute dialog than from several visitis of a website or an app. This basically mesns, that the AIs will perform qualitative user research on us.
It will leverage the collected data to provide us with much more deeply personalized content and services. This may in effect result in the Machine-led User Experience design, where we will be using interfaces composed of whole workflows designed by AI.
Should robots have rights?
Once we start talking to machines about more and more complicated and maybe even intimate topics, how will our relationship change? And what about their status as pure “things”, „tools“ or “machines”?
You might have seen the following video before, but go ahead and watch it again and try to focus on the thoughts flowing through your head:
How was watching it? Has it caused some emotions in you? If so, then what emotions? Doesn’t it resemble bullying? Those robots are artificial machines, right? Maybe they are, but now imagine the same machines covered in some material simulating flesh and animal skin to make them look a bit more living. How would these in fact violent acts look like then? Shouldn't they follow the same ethical standards as violence on animals?
From user-centered to AI-centered design
Good User Experience is an outcome of methods and techniques called user-centered design. User-centered design is all about first starting with the user and his needs, before jumping to the solutioning.
If machines will become smarter and generally more capable and if they (in form of some robots) will inhabit the real world with us, how will the world itself have to change, for them to be efficient and less error-prone?
The video below is a recording of an art performance, where the artist, using just a white paint, easily creates a trap for self-driving car.
This simple trick points out to an important fact — for some time, the so-called “intelligent machines“ won’t be smart enough to cope with all the irregularities and unexpected events of the real world. If we will want to leverage these machines for our own good, we may need to suit our world to „their needs“.
Soon enough, the input into the design of intelligent apps and services will not be only the needs of the human user, but also the needs of the machine as well. It will be a mix of the user-centered and AI-centered design.
Summary
Today big digital services, like Facebook or Google leverage AIs, which learn from the data and activities of their users. This is basically an ongoing, large-scale quantitative research ran by AI.
Each of those particular AIs is learning something a bit different though. Facebook learns how to increase users engagement through personalized News Feed stories, Google AI learns to understand and organize all information of the world and Amazon to analyze and predict transactions.
On top of of these platforms, there is an emerging layer of personal AIs, which will get much closer and more personal to us. We will be communicating with them through voice, which will allow them to learn more — machines will be effectively running a continuous qualitative research on us. This will allow them to understand us much better and it will radically change our relationship with intelligent machines.
