Sculpting Experiences with Data — Part 2

By Joshua Newnham, Lead Design Technologist — Method London

In a previous post we explored how to design for Artificial Intelligence (AI) using the fuel that drives it — data. This was framed around the need for a new way of thinking along with a need for more expressive tools to support designing truly personal and natural experiences, such as those portrayed in the movie Her.

In this post we will continue our discussion on designing with data, focusing on some of the considerations when dealing with it.

As François Chollet puts it in his book Deep Learning with Python; AI is framed as being the next industrial revolution and if deep learning is the steam engine then data is its coal, the raw material that will power our intelligent machines — for this reason we’ll use the majority of this post exploring strategies of capturing data.

Unlike a toddler that can learn what an apple is by simply by pointing at it and saying ‘apple’, our machine learning counterparts are not quite there yet. It takes a lot of data for most machine learning algorithms to learn. Even with a simple problem, you typically need thousands of examples, and for complex problems such as image and speech recognition, you require millions of examples.

To further emphasis the importance of data, in a famous paper published in 2001, Microsoft researchers Michele Banko and Eric Brill showed that different machine learning algorithms, from simple to complex, performed very similar when given enough data. The authors conclude suggesting we consider spending time and money on acquiring data over algorithm development, highlighting that data matters more than algorithms for complex problems.

data matters more than algorithms for complex problems

As designers we need to design opportunities in the journey where we are capturing data to feed the machine — let’s now explore some of those strategies.

In some products this will simply mean observing and feeding in the user’s behavior; Spotify is a good example of this, where they feed their recommendation engine with observed listening habits of their users. But in other products, this direct feedback loop may not exist in which case we need to design explicit opportunities to capture the data.

One strategy, of designing opportunity for explicit feedback is introducing a human-in-the-loop. For example, if we were designing a Conversational User Interface (CUI), i.e. chatbot, we could introduce a human assistants to handle queries that the model (chatbot) was not confident in handling; the response generated by the human assistant could then be feed back into the model, thus making it ‘smarter’ over time.

Another approach of using human intelligence is crowdsourcing it; this is the approach we have taken for one of our projects called FINE. FINE is a system created to support mental health of young children; one of its components is a virtual avatar that expresses empathy to its user.

One of the challenges we had was ‘teaching’ this virtual avatar empathy. The approach we have taken, inspired in part by reCAPTCHA and Moral Machine from MIT Media Lab, is to create an accompanying game that encourages players to teach our avatar empathy by example, then using this data to build a model is able to exhibit empathy from a given emotional state.

This and the previous idea, of human-in-the-loop, was the approach taken by Microsoft with their chatbot Tay which learned how to respond to it’s users from the conversations it was having. If you know the story then you know that its important to filter and monitor what your model learns.

In some instances, you won’t have to look far — for example, earlier this year we built a prototype for an insurance company to demonstrate how some of their services could be delivered through a conversational user interface.

One responsibility of this chatbot was to handle basic queries from their customers. Unlike the chatbot presented in the previous post that generated its response from scratch, this model was trained to find the most likely answer given a question from within an existing repository. This repository was populated using existing information that resided on your website, and documents (logs are another potential source of useful data).

Previously I mentioned how Spotify leverages explicit user interactions to feed their Machine Learning (ML) algorithm; I would like to also highlight the opportunity of data generated by implicit user interactions, sometimes referred to as data exhaust.

To better illustrate this, I will use one of our experiments we undertook to build awareness around designing with data — WordsWorth. The prototype consisted of a custom iOS keyboard that would monitor the users language habits, capturing misspelled words along with words and phrases that were classified as having a negative sentiment.

Accompanying this keyboard was a hacked Speak & Spell device, a retro educational toy released in the late 70’s to help teach children how to spell. Normally this device uses a generic and static dictionary for its words but with the accompanying iOS keyboard we delivered a personalized learning experience giving the user words that they had previously misspelled and substitutes for those words deemed negative.

Another important considering when dealing with data is handling hidden bias. Data is the breadcrumb of our past decisions and our past decisions are inherently bias. So just like in gardening where gardeners prune their garden to encourage healthy growth and flowering, as well as good looks, we have to prune our data to remove any bias and undesirable characteristics.

We have to prune our data to remove any bias and undesirable characteristics

A good example of hidden bias was highlighted earlier this year by Rob Speer from Concept Net in his tutorial titled “How to make a racist AI without really trying”; the tutorial walks through building a basic sentiment classifier using a well know dataset and the popular trained word embedding — Word2Vec, a word embedding model that was trained on the Google News dataset. The result of the tutorial is a sentiment classifier that has inherited bias towards race; for example, the phrase “Let’s go get Italian food” gets a high score of 2.3 while the phrase “Let’s go get Mexican food” gets a low score of 0.38.

Here we have only just scratched the surface of what to consider when designing with data but it is our hope that this acts as a catalyst which encourages the design community to start exploring how to design for AI, and in doing so, sharing their learnings and tools that will help shape the future and drive how we interact with computers.