How to Play Chrome’s Dinosaur Game With Your Face

Having fun with Face API and ML5js

David Yu
David Yu
Feb 13 · 4 min read
Photo by Ryo Tanaka on Unsplash

If you use Chrome you may have seen the dinosaur game.

It’s a simple game where the dinosaur jumps to avoid obstacles, which is a perfect target for me to learn how to manipulate it with my face.

Here’s the original game, if you never played it.

The Process

After watching a few youtube videos from The Coding Train, I learned that the simplified steps for getting a prediction model to work are:

If you don’t want to write the code yourself, feel free to clone my repository at the bottom of this article. I’ve split the folders into /collect /train /deploy

Other files at the root level are the code for the game itself.


Collect Data

1. Initialize a neural network with ML5js

let options = {    
inputs: 74,
outputs: 2,
task: "classification"
};
const brain = ml5.neuralNetwork(options);

inputs is set to 74 because I am storing the facial features of mouth and jawOutline.

There are 37 points for mouth and jawOutline and each point contains a number for x and a number for y, so we multiply 37 by 2.

2. Setup face API

3. Start collecting data

In the gotResults function, capture the data by calling .addData() with brain the neural network.

4. Save Data as JSON

Add a button that triggers the following, it will save a JSON file to your download folder.

brain.saveData();

Train Data

Use the JSON file generated above, train the neural network, and save the model:


Deploy Data

This is where you test if your model actually works or not.

If you did everything correctly above, you should have three files in your download folder: model.json, model.weights.bin, and model_meta.json.

Load your custom model while keeping the Face API models, so your custom model can make predictions on the result of Face API.

If your model works, it’s only the matter of wrapping the whole thing in a function with a callback.

In the callback, I trigger the T-Rex to jump whenever the event is jump.


Experiments

Thanks to ML5js. It already contains Face-API which detects facial features.

Try #1

Initially, I wanted to use my eyebrow to control the game. However, my “raised brow” face is too close to my resting face, so the model couldn’t tell them apart.

Try #2

I decided to use open and closed mouth, as it’s more obvious.

As you can see in the image above, I hit the “Collect” button when I am ready to collect the data.

The first set of data collected only contained one label or one category. When I tried to train, it gave an error about hot encoding.

Try #3

Then I decided to collect two labels:

The Open button is actually a toggle. When it’s Open, the data collected is labeled open.

When I click on Open, it switches to Normal — if I collect data, that data is labeled normal.

Finally, I click the Save button to save a JSON file of the data.

However, when I trained with 1000 epochs (1000 iterations of training), the loss didn’t drop close to zero.

Try #4

I collected more data — especially more of the Normal face data.

It successfully predicts if my mouth is open or not.


The Result



The Game

I’m a full-stack developer learning about machine learning. Thanks for reading!

Better Programming

Advice for programmers.

Thanks to TDS Team and Zack Shapiro

David Yu

Written by

David Yu

Software Freelancer based in Shanghai. More articles at https://shanghaicoders.com

Better Programming

Advice for programmers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade