Magic Leap: Initial Thoughts
As we get ready for the first year of our Masters in Virtual and Augmented Reality, one of the things we have to think about is equipping our teaching lab. Our current VR lab, which was only set up in September, is pretty well stocked but things move fast if you want to keep up to date, so we need to choose new kit for next year. This is a bit tricky as the devices we are likely to want this time next year probably don’t exist yet.
This is particularly true of the A in our MV&AR. The VR headset market is fairly stable, with the big players likely to stay big, but Augmented Reality is still pretty experimental. Mobile phone based AR is fairly mainstream now, but it doesn’t live up to our desire to create immersive experiences. We really want AR headsets.
The obvious choice right now is the Magic Leap, which I recently tested at University College London’s VR club hosted by David Swapp.
The Magic Leap has had a lot of hype due to massive investments and high levels of secrecy. When the first dev kit launched there was a lot of negative press (including Palmer Luckey calling is a «Tragic Heap»). There were a lot of very valid criticisms, but I suspect a lot of the hostility was a reaction to the hype, so I will try to write up my first impressions of the Leap for what it is: a first dev kit of a new technology, rather than trying to compare it to the inflated expectations we might have previously had.
Augmented Reality
The Magic Leap is an Augmented Reality head mounted display. It is a pair of glasses with see-through lenses that allow you to see the real world. It uses a technology called Waveguides to project 3D graphics imagery on top of the real world (very much like its main competitor the Microsoft Hololens). This allows it to integrate virtual elements into your view of the real world (the definition of AR).
Unlike a VR head mounted display, there is no screen in front of your eyes, freeing up the view of the real world, and making the headset very light. It is stand alone, in the sense that it does not have to be attached to a PC, but there is a small, disc shaped computer that you wear on a strap. Palmer Luckey says this is his favourite thing about the system, because it moves a lot of the heavy hardware off your head to over your shoulder, where it is more comfortable.
There is a small tracked controller. It looks quite like the Oculus GO control, but it has magnetic tracking, which means it has position information and you can use it to select things by touching them. Luckey is a bit scathing about the tracking, but it felt OK to me (though maybe I wasn’t in any experiences that needed good tracking).
Display
The visual quality of the display is surprisingly good. Because waveguides are a less well developed technology than screens, they can often be quite dim, particularly as they have to compete with light from the real world. The visuals I saw seemed bright and clearly visible (admittedly indoors and away from bright sunlight).
I played Dr. Grordbort’s Invaders, one of the launch titles. In the game holes open up in your walls that reveal a steampunk robotic world, from which robots fly out and attack you. The holes in the wall looked great, they were bright and looked solid, not too different from the image made by a good quality projector on a wall. It helped that they were against a blank white wall, but even the robots that flew out at you looked good. If they were against a busy background, you could see objects behind them, but in general it did very well at the hard problem that we call “occlusion”, making the 3D graphics objects hide the real objects that are supposed to be behind them.
One of the reasons for this is that the magic leap glasses are tinted, so the real world is dimmed. It’s cheating but pretty clever and effective in most situations (I will come back to drawbacks later).
The major problem with the waveguide technology that the Magic Leap uses is field of view, which is quite restricted. I remember the Hololens FOV feeling very narrow. The Magic Leap is better (I think it is about 40 degrees), but it is very noticeable that the graphics disappear if you look down, and I often found myself twisting my head awkwardly to see a particular menu. I would almost have preferred if the glasses had restricted apertures so that the view of the real world would be reduced to the same as the virtual world. This would make my interactions with the real world harder, but it would eliminate the constant breaks in presence caused by the graphics disappearing from view.
Social
One of the big advantages of AR is that it allows for effective social interaction and collaboration, because you can see the other people around you. You can even work with the same virtual objects if you are both in AR. On the other had VR shuts you out from the people around you (though VR might be better if you are collaborating with people remotely because you can all be in the same, virtual, space unlike AR).
I didn’t get the chance to do actual AR collaboration because we only had one Magic Leap available (and I don’t know if many collaborative apps are out there). I could see other people around me and talk to them, though I did feel a bit separated because they couldn’t see what I could see.
Social interaction is the area where the clever idea of using tinted glasses becomes a problem. The glasses are dark so you can’t see the wearer’s eyes, making eye contact and a whole host of social signals impossible. This isn’t an absolute killer, after all we can have perfectly good conversations while wearing sunglasses, but it does reduce one of the big benefits of AR.
If this were a VR headset, Social would be the place where I would be praising the Magic Leap for using magnetic tracking. Newer VR headsets, like Windows Mixed Reality and the upcoming Oculus Quest, use inside out tracking. Instead of using an external camera to track the headset and controllers, the controllers are tracked from a camera on the head mounted display. This is a good solution in many ways, and more accurate than the Leap’s tracking, but it has a major disadvantage for social interaction. The controllers are only tracked while you are looking at them. This isn’t much of a problem for single user experience, you just don’t notice if your hands do something strange while you are not looking at them. But in social experiences, other people can see your hands even if you can’t. Inside out tracking can screw up your gestures and body language if it suddenly stops working when your hands are out of site.
The Magic Leap uses magnetic tracking, which is also stand alone, but can track your hands when they are out of sight. Unfortunately, in AR it is less of an advantage, because you are likely to be with people in the same physical space who can see your real hands, not a tracked virtual version. That said, I think, despite Palmer Luckey’s criticism, magnetic tracking could have real benefits for VR, possibly in combination with inside out tracking. Tracking in VR is hard so combining methods is a good idea. The first VR system I used had this kind of hybrid tracking with magnetic (though it was combined with ultrasonic tracking not inside out computer vision).
Registration
The hard part in AR is not the display, it is registration: the accurate mapping of positions between real and virtual worlds to ensure that virtual objects stay at a fixed position in the real world, rather than just floating on top of it. Without proper registration, you just have a heads up display, not a proper integration of real and virtual that I would call AR. This is a very hard computer vision problem, because the device needs to accurately sense positions in the real world, without any particular markers.
My first experience of the Magic Leap in this regard was not good. We had a lot of trouble calibrating and registering the room we were in. Admittedly there were a lot of people around and a lot of equipment getting in the way of walls, but it does show that registration can be problematic unless the environment is very controlled. Having said that, I thought the interface for calibration was good: you looked around the room and the leap superimposed graphics of what it thought the walls are. Once the room did calibrate, the registration was pretty solid.
Location based AR
I personally think that the Magic Leap might initially be best for location based experiences, where you go to a particular place that is set up for a game or experience, rather than in your home.
This is partly because of cost, the headset currently costs $2295, way out of a consumer’s price range. But it is also due to registration. It will be much easier to achieve good registration and tracking in a controlled environment than in your front room.
There are also artistic reasons why this is a good idea. AR is all about integrating real and virtual worlds. That means that it is very depended on what is going on in the real world. I think it will be hard to create good narrative AR experiences that work in any physical environment (having John Snow from Game of Thrones walking around my kitchen would be just weird, I want to see him at the Wall). Location based AR would allow creators to carefully craft both the real and virtual parts of an experience which will surely create a better effect.
This doesn’t just count for fictional experiences. I’m sure location based AR would work very well in carefully crafted engineering labs, classrooms and museums.
I’m not saying that there is no place for AR apps that we use as we walk around the city, it’s just that, after the Google Glass controversy, I think they will mostly be based on mobile phones not headsets, for a while anyway.
A Great First Dev Kit
Overall I was pretty impressed with the Magic Leap and saw a lot of potential in it for immersive AR experiences. It is a first dev kit and it has flaws. It certainly doesn’t live up to the exaggerated hype, but I doubt any product could live up to that. What I wanted to focus on was the product itself, and I think it points to a bright (if tinted) future for Augmented Reality. It definitely feels much more immersive than the mobile AR applications I’ve tried.
It certainly isn’t the only option. The Microsoft Hololens was released first. I think the Magic Leap is better than the Hololens, but it isn’t a big departure, they feel fairly similar, both use Waveguide technology. Microsoft announced the Hololens 2 between when I started this article and published it. I haven’t had a chance to check it out yet, but it will probably provide some stiff competition for the Magic Leap.
There are also simpler alternatives. You can create AR by putting a camera in front of a VR HMD and combining video and graphics. This will create less good experience of the real world, but is a more standard technology so we should not dismiss the possibility it being very successful, because we understand how to build screens and cameras very well.
Overall, I’m excited about immersive AR. The first dev kits will need a lot of work before they are ready for the mainstream. Maybe the technical challenges are harder than VR. But the first steps are promising and the potential is massive.
This is part of a blog I have started to support learners on our Virtual Reality MOOC, if you want to learn more about VR, that is a good place to start. If you want to go into more depth, you might be interested in our Masters in Virtual and Augmented Reality at Goldsmiths’ University of London.
