Design for Affect: Measuring and Visualising Stress
What is Stress?
Stress is defined as physiological change occurring whenever the body needs to adapt to external stimuli. It affects several physiological processes in the human body, such as increased muscle tension, changes in skin conductance, heart rate (HR) and heart rate variability (HRV). Due to two constantly interacting circuits of our Autonomous Nervous System, the Sympathetic and the Parasympathetic Nervous System, the human body is constructed with the ability to internally balance itself and to react to extraordinary events. While the Parasympathetic Nervous System (PNS) is responsible for maintaining bodily function, such as breathing, heartbeat, brain activity, etc., the Sympathetic Nervous System (SNS) reacts whenever extra effort is demanded from our body. Sympathetic activity leads to an increased heart rate (e.g. during exercise), whereas parasympathetic activity reduces HR (e.g. during sleep). The interaction of both systems is reflected in heart rate variability (HRV) (Sano & Picard, 2013).
Arousal and Adaptability
When sensing bodily reactions to stress, two concepts can be distinguished: arousal and adaptability.
Arousal is the basic manifestation of short-term stress. It relates to increased responsiveness and alertness of our body leading to an increase of breathing, heart rate (HR), blood pressure, electrodermal activity (skin conductance and temperature) and perspiration.
Adaptability is a concept that is comparable to the concept of coping, the process of shutting off a stress response, physically and emotionally. Lack of adaptability is the main reason why stress leads to illness. Adaptability can be measured with at least ten physiological markers however most of them are not practical for a mobile application. Literature suggests Heart Rate Variability (HRV) as a useful parameter (Sanches et al., 2010). HRV measures of the intervals between successive heart beats (Malik et al, 1996).
Stress at a Cognitive Level
When talking about stress related health problems, physiological aspects of stress, but also individual attitudes towards stress must be considered. The transactional model of stress from Lazarus and Folkman (1984) first describes the individual’s perception of a stress experience as important factor influencing health. Fundamental aspects are an individual’s perception of demand and coping. This means, the way a stress itself is perceived (positive or negative), as well as a person’s perceived ability to cope with it determine whether it has a harmful impact on ourselves or not. This in turn influences short and long term effects on our physical and mental well-being (Ferreira, 2008).
It becomes apparent, that focus should be given to coping strategies. Stress is neither good nor bad per se. Frequency, lack of habituation or failure to cope with stress are the causes of potential damage. Since these are strongly subjective, self-reflection and self-awareness are key in addressing such issues. Stress-reflection has been explored as an effective method for dealing with stress in that it invites people to better connect with their internal perceptions and emotions. In order to do so, subjects need to be aware of their affective state. Hereby, biofeedback is a useful tool to foster self-reflection and self-awareness in regard to stress.
Affective Computing
Emotions were often opposed to cognition and considered feminine, spiritual and out of control, while science and cognition was defined masculine, physically grounded and controlled (Boehner et al., 2007). Picard’s investigation in the field of affective computing first connected emotion to rational cognition and made them center of scientific study.
The notion of emotion as physiologically measurable and mathematically documentable in turn led to the idea that emotion is fundamentally a natural, biological fact, something objectively observable, definable, and containable (Boehner et al., 2007, p. 277).
Emotion, like cognition, can be seen as a form of information processing and can be measured in an objective, abstract, and standard way. Picard’s introduction did substantial work in defending against the idea that emotion must be negatively valenced, out of control, socially inappropriate, anti-rational, and anti-scientific. Thus far, technologies have largely ignored emotion as fundamental to human experience, cognition, perception, communication, learning and decision-making. Often human-computer interactions are frustrating, in part because affect has been misunderstood. Affective Computing research develops new technologies and theories that encounter affect in the design of technologies to better meet users’ needs (Boehner et al., 2007).
In recent years, the interface paradigm has shifted from an informational model of helping computers to better understand human emotion toward an interactional approach of helping people to understand and experience their own emotion; including the raw elements and perceptions of emotions, the constructed conceptions and the resulting effects (behavioral or cognitive changes). The latter approach is also known as “Affective Interaction”. With this change towards an interactional approach, current HCI and affective computing research focuses on encouraging user’s awareness and reflection on emotions instead of trying to show the ‘right’ emotion (Boehner et al, 2007).
Both fields helped increase the understanding of how affect influences personal health, created new approaches to assess stress and mood, developed new emotionally intelligent systems and invented personal technologies for improving self-awareness of affective states and designing new ways of communicating affective-cognitive states.
Biofeedback
Biofeedback is “a process that enables an individual to learn how to change physiological activity for the purposes of improving health and performance” (Varvogli & Darviri, 2011 p. 77). Precise instruments measure physiological activity such as brainwaves, heart function, breathing, muscle activity, and skin temperature and “feed” information back to the user. By mirroring physiological reactions from our autonomic nervous system back to the user, the relationships between external stimuli and how the body responds to them, can empower the user to build a relationship between what they are exposed to and how that affects their bodies. The presentation of physiological information supports desired physiological changes. Over time, these changes can endure without continued use of an instrument (Varvogli & Darviri, 2011).
Design for self-reflection
“Designing is not only to make the system work smoothly and being reliable, or making a pretty interface, but to search for aesthetics that can contribute to a more motivating relationship between the user and the application (Bolter, 2004, cited in Vaara et al., 2010, p. 4).
The role of a designer is to create a layer between the raw information given by the sensors and the representation that the user sees. The Stockholm Institution of Computer Science (SICS) and the artist George Khut have made great effort in designing and testing different biofeedback systems and visualizations dealing with topics such as self-reflection, behavior change and stress.
Design Criteria
Ambiguity
It is essential to leave just enough to interpret, instead of presenting everything the user needs to know and giving no room for interaction and self-reflection. It’s like music. Instead of giving listeners raw information of the music in form of notes, information is interpreted and transformed into sound, which is much less informational yet much more pleasant (Ferreira, 2008).
A design can be ambiguous in three different ways. The first is Ambiguity in information, where information is presented in a fuzzy, incomplete or filtered way. The user has to fill the gaps of what is presented to him. The second is Ambiguity in context, where information is fully presented; the presentation context, however, is ambiguous. And the third one has to do with the user’s relationship with the design in which the user has multiple feelings towards a design: Ambiguity in relationship.
Aesthetics
Users are much more likely to use visualizations for self-reflection when those are nice to look at. Muller (2006) found that biofeedback visualizations also should be rich and meaningful.
Interpreting bio data
If you are measuring arousal (i.e. heart rate), avoid interpreting the data at all costs. Arousal is never good or bad per se, but depends on the context. Instead leave the interpretation to the user.
This does not apply for the dimension of adaptability. Adaptability has valence by definition and can be visualized.
Visualizing Biofeedback History to facilitate self-relection
Affective Diary
One of the first self-reflection devices developed by the SICS community was the Affective Diary. The purpose of this application was to keep a regular record of daily events and experiences with biofeedback as a process of embodied enquiry. The intention was to extend traditional concepts of diaries by mirroring one’s bodily experiences throughout the day. Affective Diary was targeted towards users that usually do not reflect upon themselves, but were interested in learning more about stress related issues in their lives. Hereby, biofeedback should help users to open up for self-reflection.
While working with the Affective Diary users were wearing an armband measuring arousal in form of Galvanic Skin Response (skin conductance). An in-build accelerometer measured movement while users carried a phone equipped with Bluetooth and a camera to pick up contextual data with which nearby Bluetooth devices, received SMS and pictures taken during the day could be recorded. At the end of the day, the data was uploaded to a tablet PC and were graphically represented on the screen.
The figure’s position indicated movement by lying when there was no movement and tending to a more upright position the more movement was recorded. The color indicated arousal level with red displaying high, green moderate and blue low arousal. Moreover, users were given the possibility to make notes and comments in the visualization and by that infer meaning from their day. Result show that users were really trying to understand what they had been through. A horoscope effect could be observed in that users over-interpreted the representation. The fact that users could influence the data by commenting on it arose a feeling of appropriation (Lindström et al., 2006).
Affective Health
While the Affective Diary was a device successful in using Biofeedback in order to open up for self-reflection, it was missing an important component: real-time feedback. Affective Health systems were developed with the goal to provide: 1) a biofeedback loop allowing users to see their own bodily reactions in real time, and 2) assistance for users to find both the positive and the negative stressful patterns in their own behavior. Two different designs (layer and spiral concept) were tested and evaluated in regard to user experience (Ståhl, 2009).
Layer Concept
In the layer concept biofeedback data (GSR) was “falling” down building layers of history at the bottom of the screen. In order to look back or to see details, it was possible to drag up layers. A circle in the middle of the screen pulsating according to heart beat and the background color representing skin conductance formed the real-time feedback component (see Fig. 3). A star on the right side was circulating according to the amount of movement. Adaptability was not included into this design.
The Layer Interface, however, had some fluency problems: prior states were summarized into static descriptions representing one minute of activity each. A disjoint experience between past and present was created. This fluency problem was addressed in the spiral interface (Vaara et al., 2010) Major critique was directed toward the history component, real-time feedback was acceptable. Additionally, we would assume that users might not be able to distinguish between GSR and HR with both measuring the same concept of arousal.
Spiral Concept
In the Spiral Concept real-time data was shown in the middle of the circle. The inner circle represented heart rate, the outer one galvanic skin response. Green was associated with low, yellow with medium and red with high arousal. Real-time feedback then passed into a spiral. Like the circles in the middle, the spiral had an inner and outer area. The inner one represented heart rate, the outer one GSR. Additionally, the inner spiral section became broader, when movement was recorded. The user had the possibility to change the spiral’s scale (second, minute, hour, day, week, etc.).
In the spiral concept all design criteria ambiguity, openness for interpretation, fluency and aliveness as well as providing an interactive history of prior states was successfully implemented. As soon as users became accustomed to the visualization they were better able to interpret their data and make conclusions about their life. Participant analyzed the data in terms of ‘becoming greener’ or having been red. Emotional states were mapped towards the color. In the beginning, however, the visualization was very confusing for participants. Moreover, when testing the interface, users expressed a need for future. They wished for a representation that did not end in the present, but left some empty, unwritten room for future development. The visualization should account for user’s expectations, beliefs and desire to shape their future (Vaara et al., 2010).
Visualizing real-time feedback
Heart Analogy
In the Heart Analogy a sequence of small hearts were placed on top of sticks (see Fig. 5). The color of the heart was mapped onto arousal level and the height was related to movement. Adaptability was represented by the size of the heart.
Evaluations showed that the heart analogy was too explicit and failed to open up for self-reflection. It lacked room for ambiguity and appropriation was given (Ferreira, 2008). Moreover, the representation’s similarity to a classical graphic representation was misleading. As a first impression one could think that the height represents arousal, while actually only showing movement. This representation demanded too much cognitive load by acting contra-intuitively. The user spent more time incorrectly reading the graphic instead of reflecting on its meaning.
Butterfly Effect
In this particular visualization a butterfly flew around the display (see Fig. 6). The background color indicated arousal level with red representing high arousal and blue low arousal. The location of the butterfly represented the current arousal; the trace it left showed how high the arousal was before. The butterfly’s wings were flapping according to the user’s movement. Adaptability was represented by the more or less rugged outline of the butterfly’s trace.
Although the design fulfilled some aesthetic qualities and was good in terms of ambiguity, it failed in respect to interpretability and valence. The beating of the wings and their roughness was too hard to perceive, the history path was confusing and the colors suggested a certain judgment (Ferreira, 2008).
Dot Cloud
One of the better representations was the Dot Cloud. In this representation, dots would appear in real time as older dots would fade out. Arousal in form of heart rate was represented by the dots dispersal. The closer the dots were together, the more aroused was the user; the faster the dots pulsate the more movement was recorded. The large white circle represented the user’s adaptability (HRV) to the stressor. The larger and smoother the circle, the higher was the Heart Rate Variability, indicating a healthy body reaction (see Fig. 7, left). When the heart rate was adapting well to different situations, the circle appeared smaller and in a jagged shape.
In this representation all design requirements were fulfilled. Arousal and especially adaptability were well represented and easily distinguishable. Ambiguity, fluency and openness for interpretation were given. Only the history component was very short. Stress patterns couldn’t be inferred from this short history (Ferreira, 2008).
Dots vs. background color
The dot design was the only design actually tested in an experimental setting. Participants had to do a presentation in front of a research team. Meanwhile, they could observe their bodily reactions on a display. Two different designs were tested: an adapted version of the dot design and a design purely based on background color. For the dot design, Ferreira (2008) and her team decided to get rid of adaptability and only display arousal and movement (see Fig. 8). Instead of adaptability, the strength of arousal increase was represented by the dots color. Red represented a strong increase; yellow indicated that the arousal was remaining on a relative constant level. This second version the dot representation was compared to the design in Figure 9. In the Background Color Design only arousal was shown with red indicating high arousal and blue indicating low arousal (Fereirra, 2008).
The aim of the study was to test what effect real-time feedback can haves on the user and to collect qualitative data about the users’ experience with those visualizations. Ferreira (2008) assumed an affective loop effect in which receiving the feedback of being stressed would have a negative effect on the user and induce even more stress. This, however, was not the case; participants reported that they would not look at the interface in a stressful situation. Besides, participants preferred the dot representation due to its interactivity and its lower degree of abstraction (Ferreira, 2008).
Cardiomorphologies and Bright Hearts
Cardiomorphologies is part of a series of interactive artwork by Gerorge Khut which used biofeedback technologies. Participants were seated on a chair in an empty room with dimmed light while wearing a breath sensor and a heart beat sensor (EKG). Graphic visualizations were displayed on a large screen in front of the participant. Additionally, physiological measurements were transferred into sound. Changes in heart rate and breath were analyzed and transformed into the subject’s Heart Rate Variability (HRV). Participants could see and hear their heart and breath and observe the moment-to-moment changes in HRV.
In the first draft (see Fig. 10a) a simple blue ring expanded and contracted with the breath while appearing softer and larger with slower breath rates and harder and thinner with faster breath rates. A red ring represented the heart rate. The slower the heart was beating, the smaller was the ring (Muller et al., 2006).
Some participants found the colors visually attractive, others said they became bored by the lack of variation and wished the colors would change. In a second draft (see Fig. 10b) richness was added.
George Khut aimed to give audiences an alternative insight into the way our thoughts and physical sensations are intimately linked. Participants were supposed to experiment with their breathing and thoughts while observing physical changes of their body. When participants were asked about their experience with the artwork, the majority felt the visualizations were accurately corresponding to their memories. However, difficulties in fully understanding the representation and being able to identify how physiological components were exactly represented were found (Muller, 2006).
The same representation was later tested in a clinical setting in the so called Bright Hearts Project. The second draft was used as relaxation-training game to help. children who undergo regular painful procedures, such as injection or lumbar punctures, in order to manage their pain and anxiety. In the interaction children who maintain a lowered heart rate were rewarded with sounds and visuals that respond to their efforts. This application was very useful to distract children from their anxiety and, on the other hand, gave the children some control over their pain. Researchers are currently working on the development of an Ipad application version (Khut, 2011).
The Heart Library
The Heart Library Project was another biofeedback art project by George Khut. Emotionally-mediated changes in heart-rate were displayed in form of colored dots on a large ceiling. Participants were lying under it and could observe the projection of their heart-rate on the ceiling. The image became redder in hue as their heartbeat increased and when their rate was slowing down became bluer in color.
The visualization was perceived as abstract and personal at the same time. One participant commented:
What really interested me was that if I had a thought that was self-critical — then all the dots went red — then when I said ‘I accept myself — it all went blue. I thought ‘how quick is that!’ …Just knowing that I’m capable of big things …and at the moment I’m dealing with a life-threatening illness — so that’s important for me — that I’m actually capable of stepping into another realm as well (Khut, 2011).
