Machine Learning — A magic ingredient for prototyping

Arvind Sanjeev
Aug 26, 2019 · 6 min read
Image for post
Image for post

Machine Learning is one of those emerging technologies whose true potential and applications in real-world is still being scoped because it could be applied to almost any scenario in which there is a machine. Superficially, it has been very successful in capturing our attention for a while now through celebrity face-swapping videos or deep dreaming cats from social media. Through these creative explorations, researchers and artists have worked together to democratise many of the tools used for deep learning, which now are slowly seeping into other domains, design being one of the prominent ones. In this article, we will be sharing some of our learnings from using these tools for our projects and how, as designers, we can use them as part of our toolkit for prototyping experiences.

Image for post
Image for post
Image for post
Image for post
Examples for Deep fake and deep dream.

As designers, we rely on our ability to formulate the logical relationship between input and output while working on interactive prototypes. In cases when the input data is too complex to handle or code for such as for pattern recognition or image recognition, machine learning could step in and autonomously generate the logic that will conform to the user’s expectation. There are mainly two different ways of approaching it, which depends on how you train the machine:

Supervised Learning
As the name suggests, here you manually train the machine to what you want it to recognise just like a toddler. For example, one way of doing this is by feeding a bunch of labelled images to a machine so that the next time it sees it; the machine can recognise what it is. This technique is what we will be exploring for prototyping here later.

Unsupervised Learning
This is the new kid on the block, not a lot of work has been done on this until recently. In this technique, the machine is fed with a lot of data without any labels, and it tries to cluster, organise and visualise patterns by itself. A popular example of this is GANs (Generative Adversarial Network), all the celebrity deep fake videos or deep dreaming gifs fall under this.

Image for post
Image for post
Image for post
Image for post
Pix2pix and style transfer from Gene Kogan.

All interactive prototypes have a predefined flow; they take in inputs from the user or environment and trigger different output behaviours. Translating this input data to the output is the part that takes most of the time. Delegating this part to the machine can free up a lot of our time, which can be used for polishing the overall experience. Let me show you a simple example of a model that was trained to detect if a person is smiling or not:

Image for post
Image for post
Image for post
Image for post
Wekinator demo and wekinator training.

Here, I am using a tool called Wekinator that uses supervised learning to understand complex input through the webcam to be able to trigger different outputs. The input application I am using is called FaceTracker, this tracks key points on your face through the webcam. The data from this application is then passed to Wekinator through OSC (Open Sound Control). Wekinator is then trained to detect a smile by recording a few examples through the cam; the same is done for a neutral face. Wekinator then talks to an app made in processing, which displays a smiling emoji or neutral emoji based on input. You can check out this page for tutorials on how to train your specific model. Wekinator thus lets you classify and detect anything through the webcam. You can train it to detect if there is an object in the scene or respond to people’s gestures, making it very easy to iterate and create experience prototypes.

Image for post
Image for post
Image for post
Image for post
Projects: Indirectional Lights and Objectifier from CIID.

The FaceTracker application I have used for the above example is available here: ml4a-ofx. This repository has been created by Gene Kogan to make machine learning more accessible to artists and designers. You can also find other similar experiments here: Google AI experiments. Here is one more example from Andreas Refsgaard whose model has been trained on different sounds that let you control Wolfenstein-the game using audio:

Runway ML
This is another exciting tool based on machine learning but which is more targeted towards artists, designers, filmmakers and others in the creative field. Using techniques like style transfer or pix2pix you can explore different ways of visualising your content which would have otherwise taken a lot of manual design effort. They let creators take advantage of powerful machine learning tools without having to write a single line of code by providing an app store like experience for different applications.

One example use case of this platform, as stated on the Verge:

Say you’re an animator on a budget who wants to turn a video of a human actor into a 3D model. Instead of hiring expensive motion capture equipment, you could use Runway to apply a neural network called “PoseNet” to your footage, creating wireframe models of your actor that can then be exported for animation.

Image for post
Image for post
Image for post
Image for post
Runway ML running Style-Transfer, DensePose, PoseNet and DeOldify.

From our experience with prototyping using these tools, there are a few reasons why we think prototyping with machine learning is powerful:

  1. Faster the better
    Before machine learning, to create a functional experience of something complex, you would have to analyse sensor data manually; look for patterns and hard code them in your algorithm. But now, these new tools let you click a button and train the machine what you want it to detect. Things that took a few days and a lot of manual coding effort are now replaced by a few seconds of training.
  2. Flexibility of input
    Different projects call for different needs; some concepts might rely on detecting moments through the camera; others might be based on audio. However, irrespective of which technique of data collection you want to use, you can train the machine whatever you want through examples.
  3. Minimal or Code-free
    All these tools rely on lesser coding through their graphical interfaces and let you focus more on the final experience itself. Allowing you to create customised output behaviours for a range of different inputs without having to explicitly program them, making it faster and more iterative.

The examples presented above showcase different powerful tools that can be used for executing an idea in the mind of a creator, but what if they can do more and actually shape and create the idea itself? One interesting example of this is a sci-fi companion writer from Robin Sloan. Unlike regular text editors, this can co-author a sci-fi story with you. It has been trained with a massive collection of science fiction novels from the Pulp Magazine archive, which is used to inspire your narrative through autocompletion.

Image for post
Image for post
Image for post
Image for post
Writing with the machine — Robin Sloan

This example is a precursor to how we might work with machines in the future, where they will be a part of the creative process as well and not just the execution. They will be able to augment our creativity by inspiring and co-creating with us. Machine Learning is still a black box in many ways just like how magic is to the untrained eyes. But by using tools like these, we as designers can be conscious about it and understand it better to be able to design together for a mindful future.

Image for post
Image for post

A View from Above

Perspectives about Tech, Design & Strategy from the Swedish…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store