Eye tracking, a new era for VR or just another gimmick?
Disclaimer: Technology and software discussed below is under constant development. Subjects here may change in the future but everything is correct at time of writing.
Virtual reality has become common place now as with many major entertainment or technology focused companies having something to do with the technology, whether it be in developing the software we use or the devices that push the boundaries of our imagination. The development of new hardware is currently coming faster than expected with the new mixed reality headsets that use the new inside-out tracking technology (read more about how this works here) from Microsoft based on the hololens technology. This gives us 6 degrees of freedom without the tracking stations we have become used to having. Even with these new developments, one thing that has yet to make it to the VR headsets so far is good eye-tracking hardware — yet it’s something that many in this industry believes would be a game changing addition to the field. This is supported by Tobii, the world leader in eye tracking technology, now developing tracking hardware for the Vive, Tobii have previously partnered with microsoft and alienware to provide eye tracking solutions..
Thanks to support from Digital Catapult NI, I was able to get my hands on the new Pupil Labs Vive eye tracking inserts and play around with them (As part of the Kainos Applied Innovation team, “play around with” roughly translates as “try and break”). The guys over at Pupil Labs have been working on open source eye tracking software using their proprietary hardware since 2012. Since then they have grown from a small startup doing basic eye tracking to a market leader in gaze detection and eye tracking hardware. A great benefit to me as a developer was their open source mindset, with over 4500 commits on their github for base software and a further 235+ commits on their VR unity plugin, it is easy to find out how the software works and fix any issue you have — my version ended up looking like a monster from a H.P. Lovecraft book, but more on that later.
When I started out with the eye tracking technology, I definitely did not take into account how strange it is to stare at your own eyeball for hours at a time. To the left you can see two gifs made from the 2 different ways the software detects you eye. On top is the basic 2D pupil detection, where the red dot is the centre of your pupil and the ring in the edge of your iris. The 3D detection adds a green circle onto this diagram. This is the software essentially estimating the size and position of your full eyeball. To do this it uses calculations based on pupil diameter, and average eyeball size. Using this it can then generate a 3D model of your eyeball [below], along with more data for developers to use such as the estimated eye diameter, and the generated sphere radius and diameter.
You might be wondering by now how accurate really is it? And the short answer to this is it depends. The hardware itself is very delicate, including the focal lengths of each eye. This means when you are focusing the camera on each eye, a few degrees too much either way can be the difference between 99% confidence and 80% confidence. Along with this you have the actual camera-eye distance to think about. Pupil labs have their own headset which gives a lot of light between the cameras and your eyes and a few centimetres. With the Vive inserts, we are down to almost no light passing through and around one centimetre between the camera and your eyes. Currently to use the trackers, I have to increase the lens distance on the Vive by a few centimetres, sacrificing some of the crisp images the Vive can produce at its best.
All in all, it’s about a day’s setup to get right and even with the best focus you can get, and the perfect distance from the eyes, there will still be times when the tracking will not work because someone has put on the Vive too high on their face or slightly sideward. After moving the cameras and tweaking the headset over the course of the next week I was able to get the eye trackers working 99% of the time, where the other 1% was just a user putting the headset on incorrectly.
So far it may seem like I did not enjoy using the pupil labs hardware. This is incorrect, and quite the opposite of my thoughts. I had the past few weeks to really test the limits of the inserts and I can already see the benefits this technology will have to the fields of both AR and VR. Once the placement was correct in the headset and all the issues were ironed out, the eye tracking was consistent to the point where I could actively use it throughout the day without issue or major discomfort. This is promising as even though the product is in the early stages of development, I can confidently say that it is already by far and away more accurate and useful than I originally expected.
I built a quick application to test the accuracy of the gaze detected [above]. The application generated 6 boxes in front of the user going from easily visible to around 10 pixels wide. This gave me data back on both eyes plus the gaze hits. In trying to do this, however, I ended up having to tamper with the unity plugin supplied by Pupil Labs to fix bugs and access some of the data which is abstracted away slightly. There is currently a new version of the software that makes this data more accessible. Thankfully the software is completely open source and the development community around the pupil software was quick to respond to any issues I found and suggest ways around the issue or to fix it. I ran this application 6 times, each with a new calibration and have posted the results below in table format.
Here you can see that the gaze and right accuracy are somewhat similar although on test 2 you have what seems like an anomalous value in the gaze accuracy column. Removing this value from the set of six, I find the data on pinpoint accuracy to me more inclined to what I had found throughout my usage of the inserts.
From all of this data, you can see that the gaze accuracy is almost always higher in terms of pinpoint accuracy than per-eye accuracy. This is simply due to how our eyes gaze at an object: if the object is very small, usually both eyes have a small offset. From these values we can also see that there looks to be almost no issue with seeing any of the boxes before box 5. Box 5 is configured in unity to be 0.2m wide from ~15 meters away with box 6 being 0.1m wide. These are incredibly hard to see in the first place in VR but it gives a base to scale our objects to or at least our focal points to in VR if you are using gaze detection currently. It also shows that if you’re using the tracking for anything other than specific research cases, you are probably best using the gaze point (the midpoint between your eyes).
I have found that although the tracking is good at pinpoint accuracy to a certain degree, where this technology really shines is in tracking data over time. To show this I booted up google earth VR and went to New York and had some fun through street view, all while recording data from the trackers. This output 90 frames worth of data per second over 3 minutes. Put in this context it is easy to see how impressive the pupil labs’ data algorithms are, however they only go so far. If the frame or eye confidence (how sure the program is of your eye placement) drops for one frame during calibration or during some data gathering point, it throws a 0 value. Across the full 3 minutes I only have dropped confidence 3% of the time (I count anything below a confidence level of 0.7 as dropped). This gives a 97% uptime with the gaze detection which is more than enough to pull the data we need from the eye positions over time. I found the drops in confidence were due to either movement of the HMD itself or any CPU spikes within the application.
I went on to actually graph out the confidence of my left eye camera during an application I have been building in Unity 3D. After 2 minutes of use the results are largely what I expected to be. There are some confidence drops in the centre when looking towards the right however, at no point did I look in the top left of my screen which makes the absurdly straight line going from the top left to the centre. This is somewhere I will look into further however I believe this is caused by the pupil detection algorithm when it cannot detect the pupil for a frame. In the graphs below, everything green is > 95% confidence which as you can see is the majority of the frames. Both are of the same data set.
One last thing that is worth mentioning is the heat that comes off the cameras when they are running for extended periods of time. I opened the pupil capture program and left the Vive inserts running for an hour and at 10 minute intervals I took the temperature. Over this I found that the camera inserts on board CPUs heated up to a max of 51°C and levelled out at around 46°C. Remember these cameras are just half a centimetre from your eye. This rules out extended uses for the eye inserts for me as something that hot could do some damage to the eye should it touch the cornea. There are ways around this I have found, one of which being only record when you need to however this may not work for all use cases. Pupil labs have announced new versions of the eye inserts very recently which should help with the heat concerns but it remains to be seen.
In conclusion, the Pupil Labs Vive inserts are technology worth watching and could have many uses in the future however due to how delicate and exact the configuration of both hardware and software, I do not believe they are ready for mass adoption just yet. The software is buggy (still in constant development of course) but does the job most of the time. The hardware is delicate but once you have it placed correctly it should only need minor tweaks but the position inside the Vive HMD leaves much to be desired. Just as I was writing this, Pupil Labs have released smaller cameras that will be used which may fix some of these concerns but I’m not holding my breath. I am excited to develop more for this technology and I am excited to see what incredible applications people come up with once they get their hands on them but for at least for now, I doubt we’ll see many applications using this software outside of tech demos and research projects.
