The most compelling experience you can have in VR?
One of the first demoes you normally get shown in VR is a big drop: you are standing on a small ledge over a very long drop. People show it a lot because it is a really powerful experience. Even if you aren’t very scared of heights it feels pretty uncomfortable standing there at the edge. And it feels even worse stepping out. But you can step out, in reality you’re standing of solid ground, so you won’t really drop anywhere, if you do step out you’ll probably float. Knowing that doesn’t make it any easier. Everything in your brain is telling you this is a dangerous situation and not to step out. That experience is the essence of VR: you know it isn’t real, but you can’t help but respond as if it was. That is why it is one of the great VR demoes.
If the long drop is a really compelling VR experience that a lot of people have seen, I want to talk about a different experience that is just as emotionally powerful, and much more fundamental to the medium of VR.
That experience is encountering another person, whether that person is the avatar of a real person or an entirely virtual AI.
A social encounter is one of the classic VR experiences, you can’t really know what it is like until you have experienced it yourself. It can’t be conveyed in any other medium. It’s completely unlike seening a person or animated character on screen, even in a real time skype call.
The reason is that the character isn’t looking at you out of a flat screen, the two of you are standing together in the same (virtual) space. And sharing a space makes body language work like it does in real life.
Body Language
Our conversations are made up of words, of course, but when we talk to some one face-to-face, our bodies also play an important part. We gesture, we make eye contact, facial expression and postures. All of these body language cues communicate an important subtext to the coversation. They express emotion. They can give feedback: when I am speaking I can know whether the person listening has understood me by “reading” their facial expression. They can emphasise a point, or simply manage the conversation, indicated that I’d like to speak now.
In VR this type of body language works because you are together in the same space. A virtual person can make real eye contact with you: it feels they are looking at you, not just out of a screen. If it leans towards you, you feel a real closeness, it isn’t just a camera zoom. VR is the only medium, apart from real life, in which body language really works.
Standing in front of a character in VR brings you all the emotional experiences of a face-to-face conversation, it really feels like being with another person. My collegues and I at Goldsmiths and University College London have run many experiments in which people talk to virtual characters in VR. In all cases they have the same strong reactions they have in the real world. They all know the person isn’t real, but they can’t help but respond as if they are. They have been flattered, intimidated, empathic and even sexually attracted.
Social is the Future of VR
It’s a phrase I’ve heard many times in the last couple of years: social is the future of VR. And it isn’t just because Facebook bought Oculus. It is because social interaction works so well in VR. It is because body language works.
Not that there aren’t challanges. Body language works but you have to get it right. Encountering a character without body language doesn’t feel real. Encountering a character with wrong body language is if anything worse. We, as human, subconsciously interpret body language all the time, and we will naturally interpret what some one does as a social cue, even if that isn’t really what it means (many social anxieties have been created that way). Because it is all subconscious, we aren’t really aware of how we are interpreting a characters body language. So if it is a bit wrong, we don’t just think: “oh it’s a bug in the code”, we interpret it socially, as a person behaving strangely, which we will find disturbing without even knowing why.
One of the most important things is that body language has to be responsive. Real body language is like a “social dance” we are constantly responding to the small things the other person is doing. I respond to you and you respond to me and I respond back. Just playing back a nice animation of body language isn’t going to work, the character needs to be aware of what the other person is doing.
So we have to get it right (or at least plausible). For an avatar, representing a real person, that means tracking that person’s behaviour accurately, more accurately than our current VR controllers can do. For an AI character that means simulating body language. That’s a big challenge, but it isn’t so insurmountable as it seems, and in future posts I will explain a bit about how it can be done.
I hope I’ve enthused you about virtual social interaction and made you think about including characters with responsive body language in your VR experiences. It is a step up in difficulty from a standard VR world, but it is such a compelling experience that is has to be the future of VR.
