Emoji + Image Understanding for Bots

Kip
3 min readApr 14, 2016

When we first launched Kip, we noticed people chatting images in conversation. Users sent images, emoji or stickers to Kip instead of typing a response. That’s when it hit us — these responses are a part of Natural Language Understanding, or what experts would call a linguistic register.

What’s a linguistic register?

When you’re typing a formal cover letter, you won’t use words like “lol” or “kk”. At the same time, using “by your leave” or “salutations” sounds strange when hanging out with friends. So when we say “that sounds weird,” it means that the linguistic register is wrong.

Registers are a fascinating field of linguistics, ranging from how slang is created to why people think differently in different languages.

Linguistic Registers for messaging apps

If people speak differently based on social situations,
then people speak differently across messaging apps.

We applied this theory to chat messaging. As Kip is a shopping chatbot, top incoming user chats are photos of items, brand logos and emojis. We dropped in a machine vision layer to understand our users more (Beta).

Search by Combining Emojis

This was the most fun and surprising experiment! We’d already done research with our Universal Emoji Translator and people use emojis a lot. Yes, even on productivity messaging platforms like Slack.

Unlike other messaging platforms where emojis reflect a personal state, users were sending Kip these emojis as a kind of shorthand. Like the way someone would use a 👌 instead of typing “I got it” or 🔥 to mean “I’m going to power through this work.”

We adopted this user shorthand for search. For instance, when you combine 🐱 cat and 🚌 bus:

…You’ll get results for My Neighbor Totoro’s Catbus! There’s a myriad of combinations you can get with emojis, my favorite being:

hmmm wonder what you’ll get?!?

If you discover an interesting combination, let us know!

Search by Brand

A popular use-case is when people have a brand in mind, and are looking for specific brand items. This is especially true for items associated with well-known and reputable brands. For instance, Nike with the swoosh is instantly recognizable and strongly associated with shoes and sneakers

Search for Similar Items

Another use-case where people want products similar to the photos they send. Users are looking visually similar items, but do not have a specific brand in mind.

Kip is an AI penguin that shops with you and your friends. Try Kip for Slack by tapping the button below:

Push this button

If you enjoyed reading, please recommend. 💚

--

--

Kip

We’re a Design & Tech Agency specializing in Social and AI | https://kipthis.com | @kiptalk