Vibes Tracker: My Journey of Hacking into Virtual Gatherings for a Deeper Understanding of Human Emotions

Virtual meetings can sometimes feel flat because we can’t see the non-verbal cues. So, I made Vibes Tracker to help with that. It’s a tool that understands and shows the emotions of people in online meetings.

Right now, Vibes Tracker is just a prototype. It is powered by machine learning models that can detect and interpret emotions. Currently, it only works with Google Meet. However, I’m thinking of adding support for more platforms later on. Additionally, it can be easily tweaked to target any videos rendering inside the browser DOM.

The development process began with identifying a method to inject custom JavaScript into the relevant pages. This technique is a standard method for browser extensions to interact with web pages. The added JavaScript is designed to process video frames and identify emotions.

I added some features to make sure it can handle changes in the video, such as if someone joins or leaves the meeting, or if the video size changes. Vibes Tracker can also look at multiple videos at once if there are many people in the meeting.

The two-step process of facial recognition and emotion detection employed here begins with facial recognition algorithms identifying faces based on typical patterns like the positioning of eyes, nose, and mouth. Once a face is detected, emotion recognition algorithms analyze the features of the identified face, such as the curvature of the mouth or the squint of the eyes, and map these features to a set of predefined emotions. The pre-trained weights, packed within the extension, power these models. These weights enable Vibes Tracker to determine the likelihood of each emotion based on facial features. This allows the extension to function offline, providing a fast and responsive experience.

It shows the emotions in real-time. I’ve made sure it’s light on the computer’s resources by running the main checks less often and using efficient models.

Here’s a breakdown:

• Platform-independent but currently supports only Google Meet.
• Injects some JavaScript into web pages to get and process video.
• Uses some alternative methods for platforms that don’t give direct access to the video feeds.
• Handles changes in the video with some built-in observers.
• Can work on multiple videos at once.
• Uses a two-step process to find faces and then emotions.
• Works offline and is fast because of the pre-trained data.
• Shows emotions in real-time and is light on the computer.

The best part is Vibes Tracker is available on the Chrome Web Store and is super easy to use. It doesn’t need any permissions and doesn’t need to be set up. Just add it, and the next time you’re on Google Meet, it’ll work. If you want to turn it off, just go to the extensions tab. It’s small in size (under 4 MiB) and was made to be efficient.

I think it can be really helpful. For example, in job interviews, you could get a better sense of how the interviewer feels. Or in team meetings, it might help people understand each other better.

Building this project was fun and challenging. It adds something special to online meetings. I’m looking forward to improving it and seeing how people use it to make their online chats better. It’s worth noting that for the best readings, the face should be in somewhat close proximity, and good lighting conditions are preferred. Also, it might not run very smoothly when there’s a larger group of participants present.

To get the extension from the Chrome Web Store, visit this link: https://chrome.google.com/webstore/detail/vibes-tracker/dgggclbgdbendjpaofeljedjcokehaaf