Design for Wellbeing: Emotion Tracking to Make You Calm
Breathe Deeply
With so many stressors in our everyday lives, we all need to put aside time to destress and take a deep breath. Our web application, Breathe Deeply, promotes health and wellbeing by analyzing a user’s emotions and providing a tailored music experience to help make them more content. As a user’s emotions change, Breathe Deeply changes the music selection to promote and reinforce the user’s contentedness.
Why Affectiva
As technology has become more advanced, there is increasing potential in how machines can enhance our lives. Affectiva is a service that can track and analyze human emotion by tracking a user’s facial expressions via webcam.
Breathe Deeply leverages Affectiva to analyze and identify user’s emotions to select music that will help the user de-stress and become more content. While there are various ways that Affectiva tracks emotion, perhaps one of the most interesting is through the mapping between emoji’s and emotion. They have created a compelling way to map facial expression to emoji’s, which clearly map to emotions, and Breathe Deeply capitalizes on this clear mapping of emotion in order to fully understand how a user currently feels in order to eventually help them relax.
Designing Breathe Deeply
Our Mood Board
A mood board was constructed to explore images used to provoke a calming mood, as well as the design of existing websites that have similar goals to Breathe Deeply.
Themes we gathered from the mood board included:
- Varied shades of blue: light to indicate friendliness, and dark to indicate strength and reliability
- Use of water and/or nature as a background photo
- Clean, modular interface
- Minimal text
- Sans-serif fonts
- Use of audio
Home Screen Design Sketches
Using the ideas gathered from the mood board, we sketched a paper prototype of what we imagined Breathe Deeply would include.
The sketch’s components included:
- The name of the application at the top of the page
- A modular interface that showed user their face (what Affectiva was seeing) and their current emotion (via an emoji)
- A progress bar that indicated the user’s progress toward becoming more calm
- A blurry yet, calming image as the background
Emotion Choice
In order for Breathe Deeply to achieve its goal of promoting and reinforcing the user’s level of contentedness, we had to choose which emotions the application would decipher.
We started with a wide range of emotions that included: laughing, smiley, smirk, relaxed, disappointed, scream, and rage. As we felt that laughing, smiley, and smirk included too much overlap for our application’s purposes, smirk was disregarded. Scream was disregarded as well, as it wasn’t relevant for our application’s purposes. This narrowed the scope to laughing, smiley, relaxed, disappointed, and rage.
If the application detected laughing, smiley, or relaxed, its goal was to maintain the user’s state through playing a variety of upbeat, happy audio clips.
If the application detected disappointment or rage, its goal was to promote a more peaceful mood. We aimed to do this through playing music that was not too upbeat as to annoy or upset the user more, but to instead invoke a more peaceful mood. Audio that did so was found through user testing.
User Testing
In order to find music that users would find appropriate for each mood we were considering, we performed a form of “Wizard-of-Oz testing”. This entailed polling a variety of potential users (classmates) and asking them how a range of different music would likely affect them in a range of emotional states. We performed this using a two phase test, described below.
- Understanding that rage would likely be the trickiest emotion to handle, we started there. We asked each potential user how 5 different songs would make them feel if they were angry. This was to ensure that we didn’t select any songs that showed a trend of making people angrier.
- We then followed up by asking each potential user how 5–10 different songs each made them feel. These 5–10 songs each corresponded to one of the 5 emotions we were considering, and user comments served as a test to ensure that potential users essentially agreed with our selections.
Design Decisions
An article written by a co-founder of Affectiva, Rana el Kaliouby, notes the human tendency to treat devices as we would another human and concludes by emphasizing the need to account for concerns regarding privacy and data sharing. We tend to build a level of report with our devices, and along with report, comes trust.
Because Affectiva employs a user’s webcam to monitor their facial expressions, we needed to be explicit when letting our users know they were being recorded. To do this, rather than collecting users’ facial data in the background, we included a webcam on the screen as part of the application. The displayed webcam, along with the start and stop buttons, made it very clear the user was being recorded.
Abiding by “Allowing efficient direct invocation and termination”, one of the principles Eric Horvitz outlines in “Principles of Mixed-Initiative User Interfaces” we were sure to include start and stop buttons on the webcam, along with play and pause on the audio portions of the interface. We wanted to make sure that users were able to easily invoke and terminate portions of the website as they saw fit.
We also tried to “Minimize the cost of poor guesses about action and timing”, another usability principle outlined by Horvitz. We did so by taking an average of the user’s emotions over a fifteen seconds. The average emotion was then used to determine the audio best suited for their mood. Had we changed the audio every time Affectiva read a different emotion, the corresponding audio change would be too fast-paced and jarring to the user. Our implemented average method worked to minimize both the amount of poor guesses, along with their accompanying repercussions.
Visual Design
A number of changes from the initial sketch were implemented to the final Breathe Deeply application. The final result includes four main components:
- A short explanation of the application, accompanied by instructions
- Webcam display, with start, stop, and reset buttons
- A gif to promote a calming breathing exercise
- Play and pause buttons for the audio
An explanation of the application was not included on the initial sketch, however it was implemented in the final design. As it is important to maintain a sense of transparency with the user, we were sure to include a short description of the application and how to use it.
Although the user’s current mood was included in the initial sketch, it was not included in the final application due to technology and time constraints. Given more time, Breathe Deeply would indicate the user’s level of calmness.
An anxiety-reducing gif and instructions were also included in the final interface. This component was included to lead users through a meditative breathing technique in hopes of improving their mood.
Both the webcam display and play/pause buttons for the audio were included in the initial sketch and the final application.
Many of the themes uncovered from the mood board were displayed in the final visual design such as:
- Varied shades of blue
- Use of nature as a background photo
- A clean, modular interface
- Minimal text
- Sans serif fonts
- Use of audio
Code Behind the Application
There are some important features we considered when designing the code for this application, the greatest being connecting sound with the information Affectiva records about the users. Our music implementation was through the use of the Javascript library Howler, which is compatible for all different types of web browsers and has easy-to-learn features like fading in and out music, and checking whether a song is currently playing; all of which were important features especially in connecting our implementation of sound with Affectiva recorded data.
One of the toughest things that we encountered was that Affectiva collects information constantly, and having a song change that often simply overloaded any web-browser, and would either crash the website itself, or no sound would play at all. Our solution to this was instead, we set a 15 second timer and then a song would play based on the average emotion detected in that 15 second interval. Further, since changing a song every 15 seconds can seem jarring — essentially the opposite of our design goal — we wanted to ensure there was a smooth transition, so we chose to have one song fade into another. In essence, if after 15 seconds the user was still experiencing the same emotion, the same song would continue to play, but if their emotion changed, a new song matching the different emotion would play.
We also concluded through testing that it’s quite difficult to please a wide variety of users with different music tastes, so each emotion has 5 different potential songs that can play, and our our player will randomly play a new song once a new emotion is triggered.
Final Product
Reflection
Strengths
Breathe Deeply had two main strengths:
- Concept behind the site
- Design of the site itself
The goal of the site was to provide users with a tool that uses music to help them calm down and destress. While a large part of this task included using Affectiva data to determine what song should be played, an equally large part of the project required creating a visual environment that promotes relaxation. The layout of the application was designed simply, containing only the necessary features: a brief description, the webcam with start and stop buttons, an animation to help breathing, and a pause button for the music.
Weaknesses
While the concept of the application meshed well with its visual design, we struggled with the following features:
- Audio & song/audio choices
- Feedback to user about emotion analysis
The audio itself was a bit buggy, and our analysis period of 15 seconds made for slow music transitions. Our music selection was limited to music our team members found, and even with the music we found, we were tasked with assigning an emotion to it. Some users had mentioned wanting to hear music they were more familiar with, as well having the option to listen to soothing sounds, such as rain or waves. Users mentioned they wanted more feedback about what emotions were detected, as well as an indicator about how calm they were — or at least how calm the application analyzed them to be.
Conclusion
Based on user feedback, we were successful in designing for wellbeing. Users felt the concept of the application meshed well with the visual design of the application, and the site as a whole provided users with a personal relaxation/meditation experience. That being said, the extent to which the application improved wellbeing was limited, mainly due to issues with audio playback.
I like the concept of having your own personalized meditation it could be helpful for people with anxiety — feedback from one classmate
With the open-ended nature of this design sprint, especially in the ideation portion, we were faced with the challenge of choosing an aspect of wellbeing for which to design. We were also responsible for applying the design process as we saw fit, which included ideation, user testing, and making decisions according to various design principles.
The product we created is Breathe Deeply, which is an application aimed to improve the wellbeing of its users. It promotes relaxation and contentedness by tracking users’ emotions and playing music to bring them a greater sense of calm. Breathe deeply is one of many websites dedicated to promoting wellbeing, and is indicative of the recent trend in using technology for “good.”
