Machine Learning is one of those emerging technologies whose true potential and applications in real-world is still being scoped because it could be applied to almost any scenario in which there is a machine. Superficially, it has been very successful in capturing our attention for a while now through celebrity face-swapping videos or deep dreaming cats from social media. Through these creative explorations, researchers and artists have worked together to democratise many of the tools used for deep learning, which now are slowly seeping into other domains, design being one of the prominent ones. In this article, we will be sharing some of our learnings from using these tools for our projects and how, as designers, we can use them as part of our toolkit for prototyping experiences.
As designers, we rely on our ability to formulate the logical relationship between input and output while working on interactive prototypes. In cases when the input data is too complex to handle or code for such as for pattern recognition or image recognition, machine learning could step in and autonomously generate the logic that will conform to the user’s expectation. There are mainly two different ways of approaching it, which depends on how you train the machine:
As the name suggests, here you manually train the machine to what you want it to recognise just like a toddler. For example, one way of doing this is by feeding a bunch of labelled images to a machine so that the next time it sees it; the machine can recognise what it is. This technique is what we will be exploring for prototyping here later.
This is the new kid on the block, not a lot of work has been done on this until recently. In this technique, the machine is fed with a lot of data without any labels, and it tries to cluster, organise and visualise patterns by itself. A popular example of this is GANs (Generative Adversarial Network), all the celebrity deep fake videos or deep dreaming gifs fall under this.
Prototyping deep dive
All interactive prototypes have a predefined flow; they take in inputs from the user or environment and trigger different output behaviours. Translating this input data to the output is the part that takes most of the time. Delegating this part to the machine can free up a lot of our time, which can be used for polishing the overall experience. Let me show you a simple example of a model that was trained to detect if a person is smiling or not:
Here, I am using a tool called Wekinator that uses supervised learning to understand complex input through the webcam to be able to trigger different outputs. The input application I am using is called FaceTracker, this tracks key points on your face through the webcam. The data from this application is then passed to Wekinator through OSC (Open Sound Control). Wekinator is then trained to detect a smile by recording a few examples through the cam; the same is done for a neutral face. Wekinator then talks to an app made in processing, which displays a smiling emoji or neutral emoji based on input. You can check out this page for tutorials on how to train your specific model. Wekinator thus lets you classify and detect anything through the webcam. You can train it to detect if there is an object in the scene or respond to people’s gestures, making it very easy to iterate and create experience prototypes.
The FaceTracker application I have used for the above example is available here: ml4a-ofx. This repository has been created by Gene Kogan to make machine learning more accessible to artists and designers. You can also find other similar experiments here: Google AI experiments. Here is one more example from Andreas Refsgaard whose model has been trained on different sounds that let you control Wolfenstein-the game using audio:
This is another exciting tool based on machine learning but which is more targeted towards artists, designers, filmmakers and others in the creative field. Using techniques like style transfer or pix2pix you can explore different ways of visualising your content which would have otherwise taken a lot of manual design effort. They let creators take advantage of powerful machine learning tools without having to write a single line of code by providing an app store like experience for different applications.
One example use case of this platform, as stated on the Verge:
Say you’re an animator on a budget who wants to turn a video of a human actor into a 3D model. Instead of hiring expensive motion capture equipment, you could use Runway to apply a neural network called “PoseNet” to your footage, creating wireframe models of your actor that can then be exported for animation.
From our experience with prototyping using these tools, there are a few reasons why we think prototyping with machine learning is powerful:
- Faster the better
Before machine learning, to create a functional experience of something complex, you would have to analyse sensor data manually; look for patterns and hard code them in your algorithm. But now, these new tools let you click a button and train the machine what you want it to detect. Things that took a few days and a lot of manual coding effort are now replaced by a few seconds of training.
- Flexibility of input
Different projects call for different needs; some concepts might rely on detecting moments through the camera; others might be based on audio. However, irrespective of which technique of data collection you want to use, you can train the machine whatever you want through examples.
- Minimal or Code-free
All these tools rely on lesser coding through their graphical interfaces and let you focus more on the final experience itself. Allowing you to create customised output behaviours for a range of different inputs without having to explicitly program them, making it faster and more iterative.
Co-Creating with the Machine
The examples presented above showcase different powerful tools that can be used for executing an idea in the mind of a creator, but what if they can do more and actually shape and create the idea itself? One interesting example of this is a sci-fi companion writer from Robin Sloan. Unlike regular text editors, this can co-author a sci-fi story with you. It has been trained with a massive collection of science fiction novels from the Pulp Magazine archive, which is used to inspire your narrative through autocompletion.
This example is a precursor to how we might work with machines in the future, where they will be a part of the creative process as well and not just the execution. They will be able to augment our creativity by inspiring and co-creating with us. Machine Learning is still a black box in many ways just like how magic is to the untrained eyes. But by using tools like these, we as designers can be conscious about it and understand it better to be able to design together for a mindful future.