Honest Signals

Image: ALFRED-EISENSTAEDT-WIDE-RANGE-OF-FACIAL-EXPRESSIONS-ON-CHILDREN-AT-PUPPET-SHOW-THE-MOMENT-THE-DRAGON-IS-SLAIN_I-G-27–2760–732TD00Z — retrieved from sayyes.com on 7 January 2015

A new avenue of investigation for the research project is “honest signals”.

A major work on this area is the book “Honest Signals: How They Shape our World” by Heibeck and Pentland (http://books.google.com.au/books?id=GmUXGwq8O9EC).

Background

Heibeck and Pentland indicate that this is part of a new field called “network science” which is focused on understanding people “within the context of their social networks rather than as isolated individuals”.

The use of sensors has opened up the ability to examine large numbers of humans within the context of their social networks as previously, analysis was through using simple and manual tools.

What are honest signals?

Honest signals are biologically based. In that they occur as unconsious or automatic reactions and they are not pre-planned in any way.

Honest signals are:

signals that are either so costly to make or so difficult to suppress that they are reliable in signalling intention.

Planned signals are things like smiles, frowns, fast cars, fancy clothes. We are aware of these and they are often carefully planned as we want to control how we are perceived. In that respect they are not reliable as being honest.

We need to look for signals that are processed unconsciously, or that are otherwise uncontrollable, before we can count them as honest.

The signals that Heibeck and Pentland examine are:

• influence — the amount of influence each person has on another in a social interaction, or the degree of control of one person over the conversation, reflecting the desire to dominate that conversation (Byun et al, 2011).
• mimicry — reflexive copying of one person by another during a conversation. Mimicry reflects agreement with the person or empathy (Byun et al, 2011).
• activity — the energetic state of a person. Increased activity levels normally indicate interest and excitement (Heibeck and Pentland) or involvement in a conversation (Byun et al, 2011).
• consistency — the degree of regularity or cadence of behaviour, primarily during speech (Byun et al, 2011). When there are many different thoughts or emotions going on in your mind at the same time, your speech and even your movements become jerky, unevenly accented and paced. A regular pace of thoughts reflects mental certainty (Byun et al, 2011).

Why are honest signals important?

They form a second channel of communication not just a “back channel or complement to our conscious language”. They are ancient primate signalling mechanisms. They are also associated with the underlying cognitive state of the person (Byun, Awasthi, Chou, Kapoor, Lee, and Czerwinski, 2011) and carry cues about the person’s underlying state (emotional, fatigue, or confusion). Importantly they are difficult to hide (Byun et al, 2011).

Measurement

Through examining normal conversational turn taking we are able to measure the timing, energy, and variability of the interaction. From this are able to identify the different honest signals.

They used a device called a sociometer to measure these things.

Kim, Chang, Holland, and Pentland (2008) used a “social badge” (sociometer) to collect body movement, proximity to other badges, and speech characteristics.

From body movement they were able to detect gesturing, walking, sitting, and social interactions such as body movement mimicry or rhythmic patterns.

From proximity they were able to determine relational distance and position of users, capturing and identifying face-to-face interaction.

The speech characteristics they were able to identify were speaking speed and tone of voice. From this they was able to identify social signals such as enthusiasm, interest level, persuasiveness and nervous energy.

Applications

Byun et al, 2011

Byun et al, use the social badges to identify the features of honest signals, characterise those signals, and to present this back to the person. The context examined was video conferencing.

The non-linguistic features that the social badges collected and were used by the researchers were:

• Voice — pitch
• Voice — activity
• Voice — spectral distance
• Video — magnitude of motion of the centre of the face
• Video — average magnitude of motion vectors inside the facial region
• Video — average magnitude of motion vectors outside the face
• Video — facial expression
• Turn taking — barge-in, grant-floor and suppression
• Average speaking rate
• Pitch variation
• Syllabic rate variation
• Spectral distance variation

Prosody (rhythm, stress and intonation) is closely related to the emotional state of the speaker. High variation tends to indicate the person is under some kind of stress.

As part of the study, participants engaged in a number of different staged conversations. During these conversations, the system developed by the researchers provided feedback to the participants about themselves. The feedback incorporated:

• Level of Excitement as a proxy for activity
• Level of Openness as a proxy for converse of consistency and/or influence
• Level of Agreement as a proxy for mimicry
• A speaking timeline
• Overall speaking proportion of the participant

Examining the provision of this feedback, the researchers found:

• There was variation on the degree to which participants could use the information in real time
• Most participants agreed that the system could provide value during video conferences
• Most participants found the speaking time and speaking proportion extremely useful — they found this did alter their behaviours acting as a “coach” to encourage them to speak less or a “prompt” to speak more
• Some participants liked the consistency and influence levels
• Many wished they could see the consistency and influence levels for others in order to compare
• A couple of participants liked the activity feedback

The researchers felt the user interface needed work to make it less distracting, more ‘glanceable’, and cumulative for the whole conference.

This study is interesting as an exploration of people’s response to having information that is somewhat indicative of their underlying emotional state made visible to themselves.

Kim, Chang, Holland, and Pentland, 2008

Kim et al, also used sociometric badges in order to provide persuasive feedback with the aim of improving group collaboration. The system was called meeting mediator (‘MM’).

The study examined groups working around a task. The conditions included with and without MM as well was distributed and colocated groups.

The badges collected information about:

• Body movement — from this was derived gesturing, walking, sitting and social interactions such as body movement mimicry or rhythmic patterns
• Proximity to other badges — From this, relational distance and position of participants was captured. This was used to identify face-to-face interaction and position within an indoor space.
• Speech characteristics — such as speaking speed and tone of voice. From this the researchers identified social signals suach as enthusiasm, interest level, persuasiveness and nervous energy.

The information collected was visualised to participants in the meeting/activity. It included:

• Level of interactivity of the meeting
• Balance in participation
• Speaking time of each participant

The researchers found, in terms of behaviour change:

• strong effect on speaking dynamics by reducing overlapping conversations
• compared to other groups participants with MM had significantly shorter speech segment lengths — concluded that MM increased the interactivity level
• Where the group included one or more dominant people, MM reduced the difference between co-located and distributed collaboration in terms of speech overlap. Normally there is more speech overlap in distributed groups but with MM this reduced. They speculated that this was due to the application providing visual feedback which may ordinarily be difficult to communicate within a distributed group.

This is interesting as it provides an idea of how people react in a social setting to having information that is indicative of their emotional or cognitive state made visible to everyone in the meeting. However, it seems that the information is largely being perceived around balance of contribution or involvement in a meeting rather than having a better understanding of how someone feels.

Application to the project

This field is relevant to the project, as it is examining the potential to detect and then use channels of communication that a not based in conscious language or planned reaction. These channels of communication remain important to establishing and maintaining the relationship that underlies an interaction.

However, it seems that ‘honest signals’ are a little wider than simply the emotional state of a person involved in an interaction. To be honest, I’m not really sure that this is the case. It seems likely to me that what ‘honest signals’ seem to be are those actions or responses that are based in emotions that arise from our limbic system rather than higher order emotions. To that extent they are primitive or (almost) biological, as desrcibed by Heibeck and Pentland (2010).

Given the ‘origin’ or nature of honest signals, it seems logical to look at physiological responses that are also based within the limbic system.

References

Byun, B., Awasthi, A., Chou, P. A., Kapoor, A., Lee, B., & Czerwinski, M. (2011). Honest signals in video conferencing. In 2011 IEEE International Conference on Multimedia and Expo (ICME) (pp. 1–6). doi:10.1109/ICME.2011.6011855

Heibeck, T., & Pentland, A. (2010). Honest Signals: How They Shape Our World. MIT Press.

Kim, T., Chang, A., Holland, L., & Pentland, A. S. (2008). Meeting Mediator: Enhancing Group Collaborationusing Sociometric Feedback. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (pp. 457–466). New York, NY, USA: ACM. doi:10.1145/1460563.1460636

Waber, B. N., Aral, S., Olguin, O., Daniel, Wu, L., Brynjolfsson, E., & Pentland, A. (2011). Sociometric Badges: A New Tool for I.S. Research (SSRN Scholarly Paper No. ID 1789103). Rochester, NY: Social Science Research Network. Retrieved from http://papers.ssrn.com/abstract=1789103