The Duplex problem
In the last couple of posts I’ve been talking about Jaron Lanier’s new book “Dawn of the New Everything: a journey through Virtual Reality”. In this post, I want to talk about one thing he discusses in the book quite briefly, a problem that I’ve known about for many years, but for the first time Jaron has given it a name: the duplex problem.
In my research, I’m really interested in how social interaction works in virtual reality. Most of my own work is really about interaction with completely autonomous characters agents that are controlled by the computer. But there’s also a lot of really interesting work to be done in terms of how we can set up alive conversation between two real people in VR each of whom is immersed in VR and each of whom can see some representation of the other person.
Some aspects of this are relatively solved problems: we can transmit audio and we’ve been able to do that since the birth of the telephone. But the real benefit of VR comes not just from being able to hear what the other person is saying but from the fact that VR can allow body language to work. So if we’re in a virtual environment with somebody else we can really experience their body language. We can really experience eye contact; We can experience closenes; We can experience gestures as if it was in real life.
Body language works because we both have bodies, life sized bodies, and we share the same virtual space. It works as if we were really in the same physical space as if we were two people talking in a real cafe and that is quite unlike current video conferencing technology like Skype where we’re each seeing the other person on screen not standing right in front of us.
Body language is a really powerful part of the VR medium, as I’ve said before and one that can really make social interaction feel real but actually it’s not as simple as that. Getting that body language to work is really quite complex. This is particularly true for an avatar which is representing another person. We want the avatar’s body language to really accurately reflect the body language of the other person. If it doesn’t we’re getting a really wrong impression about the person we’re talking to. If they seem stand offish or rude, because the avatar isn’t moving enough, when actually they’re being very friendly to us and that would be a bad thing.
So how do we get that body language? We need to sense what the other person is doing. We could do that by having sensors that track their their hand movements, their eye movement and their facial expressions. These could be mapped that onto a character. That could work but it’s hard to get that fully realistic.
It’s hard to really realistically represent somebody and what’s interesting is both the Oculus avatars and the Facebook Spaces avatars, that we’ve seen in VR, are not trying to be realistic at all. They’re trying to be very abstract and cartoony and that’s really why because there’s a limit to how much we can sense people’s actual body language and represent it. So if we’re getting a character that’s way too realistic much more realistic than the sensory input we’re getting about the person well we’re giving the wrong impression. The character might look really good but might not be doing much body language even though the real person is doing a lot. That could make them seem distant, or at any rate give the wrong improssions.
Another way to do that is to actually transmit everything about a person in real time. And this is it’s been a relatively old technology but Microsoft demoed a recent example called Holoportation. They capture, in real time, not just a single video image but a full 3D model of that person at every given instant using a whole bunch of different cameras. What you can do is transmit the person as a whole so you’ve actually got that person moving in 3-D in front of you. While the technology isn’t absolutely perfect, it can really create the effect of being with another person.
But there’s still a problem. If you’re both in VR you’ve both got headsets on so you can’t capture eachothers’ faces. The images that we see in Holoportation aren’t actually people in VR they’re people who are just standing in a Holoportation environment and can’t really see the other person.
We can’t have a two way conversation. Yes you could see me and get a really realistic impression of how I’m moving but I won’t see you back in the same way. Social interaction isn’t just a one way thing, it’s a two way interaction. It won’t work unless both people can see full body language.
So that’s the duplex problem: how do you get a realistic representation of yourself to another person and get a representation that person back to you both in virtual reality and in real time.
So that’s going to be one of the interesting challenges for VR over the next few years. What’s the answer? Well nobody really knows. I think one interesting thing that’s Jaron Lanier says in his write up is when he’s talking about body language and eye contact he talks about sunglasses. Sunglasses are something that stops you making eye contact because you can’t see the other person’s eyes. Now they maybe reduce the power of social interaction but they don’t stop us talking to other people. We are used to talking to people wearing dark gless and it might be that we get used to having a VR interaction which is a bit like people wearing dark glasses. Both of us are wearing headsets and we just see the other person but we see them wearing a headset which covers their eyes.
Or maybe we can do a little bit better. Maybe we can use Holoportation and also have sensors in the headset that track the persons eyes (eye tracking headsets have already been released). It might be possible to digitally remove the headset in the Holoportation model and replace it with something that looks more like glasses. Underneath those glasses you could have a 3D model of the face whose eyes are controlled by the eye tracking.
This eye tracked 3D model would be less realistic, and less well animated than the rest of the body, but that is why we need virtual glasses on your avatar. They could be transparent but a bit fuzzy, like thick glasses, so we can see the eyes but don’t notice the poor quality. Since we can get used to talking to people with sun glasses we could probably get used to talking to some one through an avatar like that with the benefit that we still can make eye contact.
That’s one solution anyway, it might work or it might not, I haven’t tried it. There are other solutions, maybe avatars will stay quite abstract like the Oculus or Facebook avatars, and we will get used to that. Maybe some one will come up with a much better solution. Who knows.
So there we have it: an interesting problem: the duplex problem. It’s not one we know solution to but I think one that a solution will emerge that is not perfect but people will get used to it. I think it is such an important problem that the way people interact in virtual reality will really be determined by the solution to this problem.
