Big Tomorrow
Published in

Big Tomorrow

Understanding How Machines Learn, Through Prototyping

by Big Tomorrow | January 23, 2017 | More Articles

Demo of The Brain by Mike Matas

Mike’s Prototype

If you watched the video, you’re now decently familiar with The Brain, a prototype by designer Mike Matas that can learn to recognize drawings and associate them with emojis. Here’s a demo of it on Youtube.

  1. Drawing Tool — Mike provides a drawing tool so users can teach The Brain how to recognize different types of drawings.
  2. Brain — The Brain itself is powered by a neural network, a machine learning approach that roughly mimics how our brains work.
  3. Perception — In order for The Brain to learn what smiles or frowns look like, it needs to be able to see drawings.

Drawing Tool

Drawing software usually works by capturing the x,y position of a cursor, stylus, or finger during a pressed state and adding those coordinates to the screen as individual dots or points in a path.

Drawing Tool by Callil Capuozzo — we borrowed his code


This part was trickier. Sifting through a bunch of computer science white papers brought us no closer to writing our own algorithm. The backup plan was to look for a neural network javascript library (Framer usually plays nice with them) that didn’t require a ton of configuration or machine learning know-how.

  1. First you structure and label input data (i.e. the things you want to train the neural network to understand).
  2. Next you do some light configuration of your neural network’s inner workings and feed the data into its input layer so that nodes within hidden layer(s) can learn to recognize patterns — a deep neural network is where you use multiple hidden layers to look for patterns in patterns.
  3. Lastly, you run test data (i.e. the things you want the neural network to interpret) back through the trained network and the hidden layer(s) determine the probability the data resembles something in the possibility space of patterns it learned via an output layer.
Diagram of a basic neural network


The overarching mechanics of the Drawing Tool and Brain components of Mike’s prototype were fairly self-evident, but how the Brain sees drawings was less obvious. We deduced two possible way:

  • Pixel Investigation — Placing a fixed-size boundary box around each drawing allows you to iterate through every pixel within that box and assign values based on the color found within each one. You then add these values sequentially to an array giving you drawing data to start manipulating.
Pixel Investigation
  • Path Investigation — Since x,y coordinates of every drawing are already captured while we draw, that path data might be sufficient for the neural network to find meaningful patterns.
Path Investigation
  • Mapping Values — Pixel data is captured either as RGB values that look something like black pixel[r:0, g:0, b:0] or x,y coordinates that look something like [x: 200, y: 133], [x: 201, y:135], ... But the neural network only understands values between 0 — 1, so we had to map ranges and flatten values.
  • Pruning Data — Since drawings are user-generated, they can be lots of different sizes. This is problematic for the neural network, because it can only understand arrays of the same length. We reconciled this issue by taking the length of the shortest drawing and pruning data from the other drawings to match. The challenging part here was to prune data that wouldn’t corrupt the neural net’s understanding of a drawing’s form (i.e. instead of just cutting a bunch of data off the end of arrays, we selectively removed data from different parts of each one).

Making It Work

Download the prototype to follow along. To run it in Framer Studio, you’ll need to import Brain.js using NPM — instructions below. From here on out, the word ‘layer’ refers to the Framer ‘layer’ construct, not to be confused with neural network layers discussed earlier.

  1. Import Brain.js Library (Brain)
  2. Drawing Tool Setup (Drawing Tool)
  3. Data Parsing Setup (Perception)
  4. Capturing Teach Data (Perception)
  5. Capturing Play Data (Perception)
  6. Neural Net Setup (Brain)
  7. Training Neural Net with Teach Data (Brain)
  8. Running Neural Net with Play Data (Brain)

1. Import Brain.js Library

Start by creating a new Framer project and saving it as brainproto.framer on your desktop. Then open terminal (fear not, we’re basically just copying & pasting) and follow Brain.js install documentation to retrieve the library with NPM — if you’ve downloaded the project, you’ll need to do this as well.

cd desktop/brainproto.framernpm install brain

2. Drawing Tool Setup

A few parts of the Drawing Tool have starting states to plug in.

3. Data Parsing Setup

We already mapped the values. Let’s make sure the arrays we’re feeding the neural net are all of equal length. To accomplish this, we’re going to wait until the user has finished all the training drawings and then find the one with the shortest array length — that’s our standard. We’ll then prune select points from all the other drawing arrays until they match the standard. Solution’s a little ugly, but it gets the job done!

4. Capturing Teach Data

Now to use all the nice functions we setup. Tell teachDraw to listen for the user to start drawing in the bottom half of the screen. Then we’ll simultaneously draw the path as an SVG and parse captured x,y coordinates in preparation for neural net training.

5. Capturing Play Data

Once we train the neural network, we want the user to be able to create test drawings and see if it can recognize them. Mike’s prototype gives the user its most confident result in the form of an emoji that pops up after the drawing’s finished.

7. Training Neural Net with Teach Data

Create a button that will appear when the user’s training drawings are finished — clicking it will initiate the trainIt() function.

Pro Tip — Open your console to watch the training

8. Running Neural Net with Play Data

Make a function that will pass the data captured from test drawings into the neural net so it can compare this to what it’s learned. The net will return a probability that each new drawing resembles a smile, frown, or tear. Populate emojLyr with the emoji that corresponds to the highest confidence drawing type before it pops up.

Closing Notes

For best results, we suggest drawing fairly slowly using a desktop browser. Be aware that you only get a single stroke for each drawing. Also, try packing as much variation as possible into your four training drawings — the more variation the neural network sees, the larger the possibility space of what it understands.

Additional Resources

We’ve really just scratched the surface here. Want to keep pushing what’s possible with machine learning and design prototyping? Check out some of the other javascript neural network libraries out there — you should be able to play with them using Framer.



Experience Strategy, Concept Development, New Product & Service Design

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Big Tomorrow

We’re a design consultancy based in Austin and San Francisco helping organizations solve complex challenges. Learn more: