Emotional Architecture
The Story of Rubix
A self devised method of cataloging the emotional state of a person using interactions inspired by the famous Rubik’s Cube puzzle.
The Current Situation
The UX Frenzy
We are continuously striving to augment the experiences of people as the world shifts towards a ‘UX’ frenzy. Today we spare so much effort toward understanding human behaviour while interacting with basically everything in the designed world. Information about human experiences has now become extremely valuable. Every company and organization wants to know how their users ‘feel’ when they interact with their products and services. They employ many research techniques to mine such information, so much so, that it often becomes intrusive and invasive. Organizations spend millions of dollars to build intricate networks that survey human behaviour and track the performance of their products. The crux of this UX data is human emotions. If we were to catalogue a person’s emotions accurately at any point in time, we would be able to fully understand the usability of anything. Furthermore, using such heuristics we would also be able to build things that provide any experience of our choosing!
However, if such a setup is possible, then why isn’t it applied?
The reason behind this deems very similar to that of any natural resource in the world. Emotions, just like natural resources (say nuclear power) are very hard to truly ‘mine’. Understanding the emotional state of a person is a complicated task that requires a lot of logistical, technological, and financial support. Hence, making emotional information a highly valuable yet inaccessible resource. Currently, the industry relies on psychologists and UX researchers to use sophisticated tools and techniques to catalogue such emotions.
Tools and Techniques you say, like what?
Emotional design tries to understand emotions of users in relation to the artefact they interact with.
Some techniques like questionnaires, interviews, self-reporting (diary studies), probes, etc. provide information that depends on the user’s ability to interpret the questions and express themselves. On the other hand, we have technological solutions like eye tracking, facial expression analysis (FEA), heartbeat analysis (HR), galvanic skin response (GSR), Electroencephalograms (EEG), and functional near-infrared spectroscopies (fNIRS) amongst others, that expose the measurements of a user’s physiology to indicate their psychology. Both these channels for emotional data collection are very important for composing psychophysiological results and measuring the emotional state dimensionally.
These discussed techniques have carried us through the advent of UX research till now, and have proven to be extremely valuable assets. However, their scope has been limited due to several lacunas.
Need for Innovation | Lacunas in existing methods
The physiological data collection tools discussed previously have several gaps. Firstly, these methods require financial support that cannot be afforded by the majority of researchers and designers on their own. Despite the existence of organisations that offer both hardware and software solutions, the prices of these tools and equipment are not accessible for budget-based research. iMotions, Noldus, and Emotiv to name a few are companies that offer such products.
Research in the lines of open platform devices (hardware and software) that allow accessible replication and improvement of physiological data tracking is very limited.
To top it off emotional data analysis along with psychophysiological data collection is a field of severe complexity in terms of understanding. There is a large level of cognitive processing and predefined subject proficiency required in order to successfully conduct such kind of research. This acts as a barrier for many UX designers that wish to bring about emotional enhancement in their work but are unfamiliar with the technical know-how of complex equipment.
On the other hand, if we were to look past the ‘machinery’ towards more manual information collection techniques, we are faced with research techniques that completely rely on the subject’s ability to articulate their emotional state. This raises several questions on the accuracy and reliability of this information.
Another interesting factor that has often been overlooked when it comes to emotional data collection, is the factor of emotional dependency. Emotions aren’t always induced or felt in terms of absolutes. The intensity of a particular emotion felt is often enhanced or diminished depending on the intensity and nature of the previous emotion experienced. Furthermore, emotions that lie on congruent palettes of positivity or negativity have the capacity to affect each other severely. For eg. Fear and disgust often have the capacity to induce and even enhance the feeling of anger. Hence, if a person feeling angry at one moment is scared at the very next, the level of fear felt will be much higher and more intense than that of starting from a neutral palette. This phenomenon has been discussed in this study as the occurrence of ‘emotional dependency’. Existing emotional data collection methods that rely on the ability of the user to articulate his/her emotional condition often neglect this very important aspect of ‘one emotion affecting the other’.
It is at this wicked juncture that Information Architecture becomes a powerful tool for a possible solution. IA allows us to find our own way, in our own time by considering our own expectations, experiences, and situations. If we treat emotions as the ‘information’, the entire conundrum becomes a simple IA problem. By attaching relevant appendages and annotations to emotional information and arranging it logically, we can point toward a direction that solves this problem.
As a nascent architect of the world of Information, I have taken a jab at the aforementioned arrangement. Generating another technique of emotional data collection slightly skewed towards the manual end is Rubix.
Phase 1: Scoping the space out
How much can the user do?
The next challenge in this adventure, was to channelise a solution which was not just theoretically sound but also stood strong under the scrutiny of other required aspects discussed previously. The gaps that were found in the existing state-of-the-art methods and tools were to be filled. The first and foremost was the aspect of accessibility. Heavy equipment had several aforementioned features that were barriers to many researchers. Hence, in order to eliminate all these barriers, the best approach was to eliminate the need for difficult and highly technical knowledge in the form of tools and equipment.
However, this meant that in order to generate feedback, a clear reliance on the abilities of a person to articulate his/her emotional state was required. There is very little literature that talks about the proficiency of a person to enumerate his/her emotional state. Hence, the first phase of experimentation was initiated with the same goal.
The primary goal of this phase of experimentation was ‘generating a stimulus and response base’ among various participants and studying their ability to accurately give data feedback about their emotional experience’. The experiment was a simple two-step process.
- The first step consisted of a channelised discussion regarding emotions and the participant’s understanding of UX.
- The next step was a simple task which involved the participant filling in responses based on certain stimuli given to them.
Two screen bases were used for the same for each participant. One was the interface, wherein the participant would fill in his/her details followed by a ‘Google Form’ based questionnaire in reference to the stimulus provided. The second screen was used to provide the stimuli.
The Stimulus
The stimuli set provided was a video. This video was an amalgamation of 9 different short clips consisting of shorter videos of different genres, aimed at generating a vast and varied emotional response. After each clip section was shown, the video was paused automatically for 1 min and the participants were asked to fill in their responses in accordance with the footage they had just seen. This way, the results gathered were immediate and unfiltered, giving the most accurate output. Similarly, after 9 such short clips the participants were asked about their overall experience. The entire duration of the experiment was 30 minutes. With a breakage of 5–7 minutes of initial discussions followed by a 25-minute video task.
The 9 clips that were shown were chosen and filtered specifically to have strong emotional responses. Some of the content was slightly disturbing as well. All the footage shown was under the consent of the participants.
Since the clips were hand selected, each clip had a target emotion(s) that it was supposed to induce. The responses of the participants with reference to pre-defined targeted emotions were compared and the level of accuracy was studied. The following figure shows the clip number in reference to its targeted emotion. The contents of the exact video footage could not be disclosed here due to copyright concerns. However, due credits were given and permissions were obtained before their usage for the same.
The aforementioned visuals represent the colour palettes used for data visualisation subsequently. In the latter graphic, divided sections of colours meant the existence of multiple emotions through a single video clip. The induction of these emotions was also in the order of representation from left to right.
The experiment was conducted in pairs, amongst a participant pool of 30 undergraduate design students and faculty members of the Delhi Technological University, New Delhi, India and the setup for the same was established inside the centre for Industrial Design and Ergonomics, Department of Design, Delhi Technological University.
The entire process for the first phase spanned over 30 days and the results were noted and novel insights were drawn. A comparative analysis was carried out which showed significant results showing resemblance and congruence in the expected emotions and the articulated emotional response recorded by the participants. The subsequent figure represents the same through graphical visualisation. The colour palette used in the following visual is in reference to the palette discussed previously.
The palette-based doughnut charts represented above clearly showed a difference in some of the recorded emotional responses. Hence, an evaluation of the amount of deviation or difference in expected and recorded responses was necessary. The following graph is a visual representation marking the deviation area and the variance area of the recorded data in comparison with the expected data readings. The percentage values were averaged in cases of multi-emotional stimuli and the readings were noted henceforth.
It can be noted clearly from the graph above that the deviation in terms of emotional conditions articulated by the participants versus the expected emotional outcome of each stimulus is very less. Only under the constraints of severe contextual barriers along with major complexity and duality of emotions felt by the participants in stimulus number 8, there were minor discrepancies in articulation and projection of data.
Based on these results a clear idea was generated in terms of the ability of various users to successfully represent their emotional conditions quantitatively. However, the level of accuracy was a concern. Here, Hick’s law, commonly used for UX practices, came into the picture. It states that as the number of choices presented to a user increases, the cognitive load increases as well, which ultimately leads to a faulty or error-prone response.
Recalling our previous discussions on the existing lacunas in the present emotional tracking tools, it was evident that lesser and basic choices were the way to go. This was done in order to reduce the cognitive load faced by the participants and generate an accurate direction gauging their emotional condition successfully and easily.
With this outcome, a clearer approach was generated, guiding the project into the second phase. Here, the primary focus was to develop and design an accurate solution based on the findings of the aforementioned literature reviews and experimental studies.
Phase 2: The Rubix Method
Colours and Cubes
The Rubix method has two dominant elements as its backbone, they are colours and cubes.
Emotions and Colors
Human emotions are often categorized into six basic forms namely Joy, Sadness, Fear, Disgust, Surprise, and Anger. These primarily define the spectrum of human emotions in a larger, broader sense. In today’s ever-changing world, it is very important to keep a holistic track of a user’s needs and wants. Maslow proposed that human needs can be organised into a hierarchy that ranges from base-level needs such as ‘food and water’ to top-tier concepts such as ‘self-fulfilment’. The notions of self-fulfilment emerge from the emotional comfort that a user experiences whenever he/she interacts with any product or situation. When we talk about psychological comfort, the kinds of emotions any user experiences are of utmost importance. The subsequent elements dictate their experiences depending upon the kind of emotion they induce. Hence for studies in the fields of UX research and design, a specialised understanding of various emotions and their effects is required.
In order to measure emotions dimensionally, it is very important to understand the root factors that affect a person’s psychology. Colour is often associated with a person’s emotions. Colour may also influence a person’s mental or physical state. For example, studies have shown that some people looking at the colour red resulted in an increased heart rate, which then led to additional adrenaline being pumped into the bloodstream replicating the physiological response of anger. Such kinds of responses can be observed around each colour. The following figure is an attempt to chart out the primary emotions in accordance with the colours that induce them.
Emotions and Cubes
The induction of different emotions is not just limited to colours in terms of their physical visualisation. Albert Mehrabian, a psychologist and researcher proposed a 3-dimensional visualisation of emotions known as the PAD model of emotions. Wherein P signified Pleasure, A signified Arousal and D signified Dominance. All emotions were mapped out inside the coordinates belonging to these three axes.
Following the lines of such 3 Dimensional visualisations of emotions, a cubical representation of different examples of emotions and their orientation was discovered. The adjoining image is one such representation inspired by the works of Brent Jason Lance. It is from this specific visual imagery, a newborn concept of a ‘cube-like’ or ‘Rubik’s cube-like solution establishing emotional context was initiated.
The Rubix Architecture
How it Works
A detailed method was generated that assigned specific emotions to specific colours of the Rubik’s cube. The colours were carefully assigned in accordance with the emotions they induced. The same can be tallied in reference to the emotional colour palette used previously. As an exception to the palette before, the colour ‘Yellow’ was assigned the emotion of ‘fear’ as opposed to ‘Black’. Furthermore, the colour ‘White’ was added in order to represent the ‘Emotional Neutrality’ or ‘Surprise’. This was specifically done to keep the entire palette in sync with a Rubik’s Cube’s identity.
After the emotions were assigned to their subsequent emotions, the order of rotation and their placement were the targets. The six emotions were mapped out in two different axes of rotation. This was done with reference to the amount of positive and negative biases they possessed. Happiness and Sadness were emotions that lay at opposite ends of the emotional spectrum and were arguable ‘opposites’ with respect to each other. Hence, Neutrality was mapped out at the centre of their junction axis. The other axis consisted of Disgust, Anger and Fear placed under a similar trajectory.
Another important aspect when it comes to emotional data tracking and analysis is the intensity of emotion experienced. The accuracy of the data collected is dependent on the scale on which the intensity is measured. Understanding the novelty of this field of research in terms of existing solutions, it was necessary to find an initial direction that successfully gauged the experience of the participants. Hence a three-point Likert scale was used with values from Low Intensity, Moderate Intensity and High Intensity. This was further in accordance with the 3 x 3 matrix used in a cube.
Each emotion along with a particular axis was also assigned a definite order, the axis of rotation and sequence of rotation. All these aspects were interlinked and helped result in an accurate visualisation of the emotional state of the participant via the matrix. The same has been discussed in detail subsequently.
The matrix has three parallel axes for each emotional paradigm. These are used to define the intensity of the emotion experienced. The same has been visualised in the following graphic.
The visualisation process is defined by the following rules :
- Happiness, Sadness and Neutrality are defined on the horizontal axis (X-axis) (L to R)
- Anger, Fear and Disgust are charted on the vertical axis (Y axis) (T to B)
- Each axis has one term of rotation. If an emotion on the same axis is experienced again, the rotation shifts to its parallel axis.
- The intensity of an emotion faced defines the number of parallel emotional axes that are turned in one term.
- The rotation of the parallel axes is sequential and the subsequent parallel axis will only change, if either the intensity of emotion is high or another emotion from the same parallel axis has been experienced again.
Complicated right?
The following is an example of the visualisation through Rubix explaining the step-by-step process for the same.
Phase 3: So, Does it Work?
Of course it does, but don’t take my word for it, try it out for yourself.
The Rubix Test
Since the matter of psychological data collection and analysis was at hand, it was necessary to introduce the existing psychological data collection methods and observe the results when both solutions were used in tandem.
The coherence in the resultant data would verify and solidify the hypothesis on which TREI functions. The graphic above represents a small clip from the beta tests. It shows a brain mapping visualisation of a person's mind when they are bombarded with stimuli. Based on the concentration, type and location of the waves (alpha, beta, theta, and gamma) we can predict the emotional state of a person. The visuals are mapped using an 8-channel Emotiv EEG device.
If we put the EEG data discussed above in parallel with the data collected using the Rubix method, we can conclude the success rate of the new method based on the level of coherence between these information sources.
Where we are now
After gaining positive results from the beta testing phase, Rubix is moving towards the primary experimentation phase. Here’s a short recruitment teaser/trailer for the same. The content of the video is an edited amalgamation of various creative pieces designed by Unbabel, Pé Grande, and Lais Tissiani Dutra.
If you are interested to know more about the tests and/or wish to get involved with the experiments, feel free to reach out to me at akundu@umich.edu
To Sum It Up
World IA Day
The Rubix project was presented at the World Information Architecture Day 2023, Ann Arbor at the Michigan League, University of Michigan, Ann Arbor. The event was sponsored by The Understanding Group (TUG)
Watch the full talk amongst other enriching presentations at the World IA day, on the youtube channel or the live stream page.
Conclusion
Rubix is an amalgamation of several attempts, at emotional data collection through repeated self-devised experimentation and experiential research. The journey was a persistent effort with different challenges throughout. Both phases were filled with severe restrictions posed by the waves of Covid Infections in India, the second phase was specifically coupled with various logistical issues ranging from lack of equipment to scarcity in connectivity.
In conclusion, The Rubix Experiment is a long-drawn effort to bring about groundbreaking changes in accessibility and usability factors for researchers, designers and psychologists throughout the world who are trying to understand ‘experiences’.
Looking Back: Note
It must note that Rubix is a nascent attempt to visualize complex information using architectural techniques. It is not a definitive alternative to existing research methods. The aim of this study is to provide a novel direction for emotional data collection using orientation and arrangement of difficult data. As a work in progress, I acknowledge that the project has a long way to go and is far from a finished solution. However, I believe, Rubix as a study, fulfils its purpose if it initiates a discussion and dialogue toward the importance of IA in the field of user experience which spans further than the built and/or designed environment.
Acknowledgements
A heartfelt thanks to Prof. Partha Pratim Das, Delhi Technological University, and Prof. Daniel Klyn, TUG, University of Michigan.
This project is a solo venture that would not have been possible without the immense support and guidance of these special people. I would also like to give a shoutout to my amazing friends and family for their continuous support and motivation.
As an independent project, the Rubix initiative is actively seeking sponsorship and logistical support in order to explore the true potential of the concept. If you are interested please reach out to me at akundu@umich.edu for the same.
Hope you had a good time reading.
Stay Tuned for Phase 3.
Thank You :)
