Playing PokemonGo, only with folks wearing a Levis t-shirt

Photo by Bruno Vaz for the amusing NIT article on people wearing Levis t-shirts.

There’s a disturbing trend going on right now (2018). I suppose it’s a global thing? But more specifically in Portugal everyone is wearing the same Levis t-shirt!

I mean eeeveeeryoooone. 🐑

If you are in downtown Lisbon or if you happen to stroll around one of the main shopping malls, there are moments where, at any given moment, you’ll literally have a Levis shirt in your eyesight. There’s no escaping it!

Portuguese comedian Diogo Batáguas joking: “I agreed to drink a beer bottoms-up each time I see a Levis t-shirt at Alive (music festival). Imma puke before 9.”

Never mind the why (social identity, group belonging) or how (Instagram influencers?). The question is — how can we have some fun with this?

What if we had an app that recognises a Levis t-shirt and, each time you see one, you’d take a photo like PokemonGo and earn some points?

We could call it, like, Levimon (get it? 🙊)

Defining the features

  1. Should be able to take photo with a phone
  2. Should be able to identify whether or not the person is wearing a Levis shirt
  3. Reward points to the player

Initially, my idea was to use the tracking.js library to track the red from the t-shirt, then isolate it in a square with a standard size using image manipulation library fabric.js.

We’d then use synaptic.js to build a Neural Network that could identify the brand.

In other words, a ton of work! First, you’d need to handle standardizing the photo to extract the brand area. Moreover and as importantly, you’d need to create a good training data set for the NN to identify the t-shirt.

That’s too f*ing long for a quick afternoon hackathon! Plus, there are some variations of it with blue shirts instead of white; some others look like a Pepsi logo. So, interesting challenge, but probably s*itty solution.

Hold on, how ‘bout using Clarifai? Those guys have some solid computer vision solutions, including identifying brands. On top of that, they have a free tier providing 5k transactions/month, ideal for our test case. Bingo! 💰

*Actually* building the app

The app itself is a standard VueJS app created using vue-cli. The whole code can be found on Github.

Here are the interesting code nuggets:

Accessing the device’s camera

We do that by using the accept and capture parameters on a regular input element, such as:

// Within Shutter.vue component
<input id="cameraInput" type="file" name="camera" accept="image/*" class="image-input" capture @change="processImage($event)">

Reading the image & passing it to Clarifai

The Shutter component simply passes the whole event on input change. App.vue handles the rest. We call the identifyBrand method when imageData event is triggered.

//App.vue
<template>
<Shutter
@imageData="identifyBrand"
/>
</template>

To read the content of the image we employ the FileReader API and use the onload callback to then feed the data back to Clarifai, such as:

<script>
const Clarifai = require('clarifai')
// Initialize Clarifai instance using your API key
const clarifai = new Clarifai.App({ apiKey: process.env.VUE_APP_CLARIFAI_KEY })
// Initialize our file reader
let reader = new FileReader()
...
methods:{
identifyBrand(imageData){
this.hasMatch = null
let foundBrand = false
    reader.onload = (e) => {
      //  We only want to pass the Base64 encoded string to Clatifai
let img = e.target.result.replace(/^data:image\/[a-z]+;base64,/, '')
      clarifai.models
// Use the right model to identify brands and pass the image as base64
.predict(process.env.VUE_APP_PREDICT_MODEL, { base64: img })
.then((r) => {
if (r.status.code === 10000) {
if (r.outputs.length > 0) {
            //  Check if Clarifai has returned any matches
if (Object.keys(r.outputs[0].data).length > 0) {
              //  Clarifai returns the multiple places with matches in the image
r.outputs[0].data.regions.map( el => {

// Bulletproof. Just looking for 'levi' match in the name value
if (el.data.concepts[0].name.toLowerCase().match('levi') !== null) {
// Only count the match if the confidence score is above .9
if (el.data.concepts[0].value >= 0.9) {
foundBrand = true
}
}
})
// If there's a match, change this value to then display a success/error dialog
if (foundBrand) {
this.hasMatch = true
} else {
this.hasMatch = false
}
} else {
this.hasMatch = false
}
}
}
},
(err) => {
alert(err)
})
.catch(err => {
console.log(err)
})
}
    // Load imageData  
reader.readAsDataURL(imageData)
},

...
</script>

Testing it 🍄


And there you go! A very rudimentary way of having some fun-with-friends: “Hey, who can spot the most Levimons out there? — Ah there’s a ton of ’em here!”.

Here’s the live test of the app:

Hunting for Levimons in Lisbon.

After walking for 1h, I spotted at least 20 people wearing it! 🙈

Most of them were too far or it was just to awkward to snap a picture hence a lot of missed opportunities.

Regarding the accuracy of Clarifai, unfortunately most of the times it wouldn’t get a match (#sad), even on very up-front & close pictures (lightning condition? Angle? Ripples in the shirt’s fabric?) so never mind my score..

Keep in mind though that this demo only keeps track of the score using Vuex, so no persistent storage. If time permits, I may have some fun in the future plugin in Firebase, just for the kicks of it.

Besides storing to a DB, some other nice-to-haves would be:

  • Streaming the camera image/video directly to Clarifai and have it more real-time;
  • Blur people’s faces automatically using tracking.js;
  • Adding some geolocation data to it;

Links