Started From The Bottom, Now There’s Beer

When beer meets Machine Learning

Wouldn’t it be cool to just be able to walk over to a bar and not have to fight for attention to get your drink? With SmartBar, we’re quenching your thirst the smart way. All you need to do is take a selfie with the app, so we can recognize you and pour your drink. And this is where our latest project with Wirecard comes in to combine two things we are quite fond of: beer and machine learning.

Biometric Payment

Wirecard and YND have been working together on innovative projects in the fintech industry. It was very nice of them to give us a sneak peek into a product that’s still under development.

It consists of a neat little hardware component, that packs biometric identity recognition, product detection, and behavior analysis technology that can be embedded in your next IoT project to enable seamless payment experience. It packs some serious computing power, pre-developed advanced neural networks, and integrated ML technology in order to enable the next big thing in payments — secure checkout without the need of you getting out your wallet or even talking to a cashier!

We took this bad boy for a spin and used it to hack together a beer dispensing machine like no other.

Here’s how it turned out:

Knowledge On Tap

Looks neat, right? But I bet you’re wondering what’s the tech behind this magic machine.

1. First, you make a selfie on a companion app, the picture is then fed into the system for reference and analyzed for so-called landmarks (position of eyes, eyebrows, nose, jawline and other specific facial features).
2. The analysis/recognition of the facial landmarks is performed using a machine learning based model that has been trained on publicly-available photos of celebrities and popular people. At some point, the system is able to recognize familiar faces by itself and you only need to correct the results to improve the model.
3. According to (average) landmark positions, a unique “vector” describing a given person is created which essentially contains distances/proportions of landmarks in relation to each other.
4. Those vectors end up (in a VERY encrypted fashion) on the chip hooked up to the beer dispenser’s video camera which is positioned to record people’s faces as they stand in front of the machine.
5. As people approach the machine, the same model is applied on the chip to analyze each video frame (in real-time) and detect faces (with landmarks) present in the video frame.
6. Those vectors are compared against the securely stored vectors from selfies of people registered in the system. If the difference between vectors is relatively small, it’s safe to assume that the face in the video frame matches the person’s selfie in the database.

Live drinks counter

The whole process of video frame analysis, face recognition, and face detection are computationally expensive. It used to require very powerful machines (with multiple cores) so that several frames can be processed per second. To pack all this processing power on a single chip is nothing short of a miracle. It was made possible only recently with developments in GPUs used by graphics cards for high-end gaming (OK, not only for that but it’s nice to think that hours spent on Call of Duty finally paid off!).

The machine learning aspect is not the only cool feature of the SmartBar. It also allows for automatic drafting, thanks to a customized BottomsUp beer dispensing system. You simply push down the cup on the allotted slot (which will light up with the color matching your order), and it will then automatically fill up from the bottom. This will speed up things and make it a lot easier for multiple people to be served at the same time.

Let Emojis Do The Talking

Now that we’ve got the tech side covered, it was time to start thinking of the look and identity we wanted for the app. With this project, we wanted to make face recognition technology more approachable by presenting it in a fun, casual way. The technology behind the whole system is very complex, but we didn’t want the people using the app to get overwhelmed by that. We needed to onboard the user to the system and explain the purchase flow in an easy way, so they will be curious to give it a try. The fact we’re using his/her image shouldn’t be scary and keep people from trying. In the end, if someone wants a drink, it should be fast! So the whole point was to make the check-out at the bar seamless meaning we had to prevent users from making mistakes, like taking a selfie in bad lighting.

Here’s how we did it:

• We adopted Wirecard’s color scheme and big fonts in the onboarding to make the app look friendly but reliable.
• Using emojis are a great way to break the ice and convey a message quickly.
• The most important part of the flow was, of course, the selfie. First, we had to explain to the user why we need it (since most people are not really used to “paying” with their faces yet). And then, we had to make sure the user takes a clear photo which allows us to recognize him later.

After all, the onboarding is a visual process, so using emojis makes more sense in this case. The animations catch the user’s attention while showing the selfie instructions directly over the dimmed camera image.

For the demo, we decided to offer the user two options: beer or lemonade. Instead of a grid with items to select, we opted for a more engaging purchase flow, while still keeping it really simple. Swipe to switch between drinks, tap to select and confirm the order. Et voilà!

SmartBar Goes Live

So once the kegs were cold and the screens are hooked up, it was time for the big reveal of the first prototype at DLD Berlin. Wirecard hosted their own space where people could mingle and network in between talks, and place an order at the SmartBar to get their drinks.

So, what’s next? If you didn’t get a chance to try SmartBar at DLD, it will be set up at ITB in Berlin in March 2018 for more drinks.

We also know that Wirecard is working hard to finalize its latest products, and to allow support for even more complex use cases such as product & customer behavior detection and more. So stay tuned!

This post was written by Nicky Jackie Delphin, Head of Content at YND. In need of some brain power? Feel free to reach out to us via hello@ynd.co with questions about your tech projects.

All set up & waiting for the crowds

This story is published in The Startup, Medium’s largest entrepreneurship publication followed by 293,189+ people.

Subscribe to receive our top stories here.