Welcome to the next installment of my VR blog series. In previous VR blogs we’ve considered the importance of clear focus to a VR experience, as well as the essential requirement to keep ‘motions to photons’ latency below 20ms in order to avoid unnecessary visual discomfort (and vomiting). This time we’re going to look at eye tracking and the impact it could have on the way we use VR in the future. Eye tracking is not new – people have been doing it for nearly twenty years – but head mounted displays for VR could be the catalyst technology needed to unlock its true potential.
One of the aspects of VR that is still presenting a challenge is how to provide a quality user experience when navigating a VR environment. Current systems have a couple of options: The Samsung Gear VR uses a control pad on the side of the display to allow you to press and hold to move or tap to engage with objects. Google recently announced they will release a motion-enabled remote controller for their Daydream VR platform later this year and all tethered VR systems have fully-tracked controllers that mirror your hand movements in the virtual world. Alongside these there’s also growing interest in making more use of your eyes.
Eye tracking is currently getting a lot of hype on the VR circuit. The ability to follow the path of your pupil across the screen has wide ranging uses for all sorts of applications from gaming to social media to shopping. We don’t necessarily have full control over our eye movements as our eyes function as an extension of our brain. This unconscious motion is very different from how we interact with our hands so there is work still to be done to design just the right user interfaces for eye tracking. How, for example, do you glance across something without accidentally selecting it? Just think of the dangerous spending possibilities of selecting items to add to your cart simply by staring longingly at them!
Several eye tracking solutions are emerging from companies such as The Eye Tribe, Eyefluence and SMI, as well as eye tracking headsets such as FOVE. At GDC 2016 MediaTek were able to demonstrate eye tracking with the Helio x20. In all cases the path of your vision is minutely tracked throughout the VR experience. The only calibration typically required is a simple process of looking at basic shapes in turn so the sensors can latch on to your specific eye location and movement. This suggests eye tracking could be easy to adopt and use with mainstream audiences without specialist training. The first use for eye tracking that springs to mind is, as usual, gaming controls and there have indeed been demos released using modified Oculus and Samsung Gear VR headsets which use a built in eye tracking sensor to control direction and select certain objects simply by focussing steadily on them. FOVE have also shown how a depth-of-field effect could be driven from the area of the scene you are looking at, to give the illusion of focal depth.
An additional potential benefit of eye tracking in VR is the ability to measure the precise location of each eye and use it to calculate the interpupillary distance (IPD) of the user. This measurement is the distance between the centres of your pupils and changes from person to person. Some VR headsets, such as the HTC Vive, provide a mechanical mechanism for adjusting the distance between the lenses to match your IPD but many more simply space the lenses to match the human average. Having an accurate IPD measurement of the user would allow for more accurate calibration or image correction, resulting in a headset that would always perfectly suit your vision. Your eyes can also move slightly within the confines of the headset. Being able to detect and adjust for this in real time would allow even more precise updates of the imagery to further enhance the immersion of the VR experience.
Eye tracking allows the view to update in real time based on exactly where you’re looking in the scene
Foveated rendering is a power saving rendering technique inspired by the way our eyes and vision work. We have a very wide field of vision with the ability to register objects far to the side of the direction in which we are looking. However, those images and objects in the edges of our field of vision do not appear in perfect clarity to us. This is because our fovea – the small region in the centre of our retina that provides clear central vision – has a very limited field of view. Without eye tracking we can’t tell where the VR user is looking in the scene at any given moment, so we have to render the whole scene to the highest resolution in order to retain the quality of the experience. Foveated rendering uses eye tracking to establish the exact location of your pupil and display only the area of the image that our fovea would see in full resolution. This allows the elements of the scene that are outside of this region to be rendered at a lower resolution, or potentially multiple lower resolutions at increasing distances from the focal point. This adds complexity but saves GPU processing power and system bandwidth and reduces the amount of pressure placed on the power limits of the mobile device, whilst your brain interprets the whole scene as appearing in high resolution. This therefore allows headset manufacturers to utilize this processing power elsewhere, such as in higher quality displays and faster refresh rates.
The High Performance range in the ARM® Mali™ family of GPUs is ideal for the heavy requirements VR places on the mobile device. Achieving ever higher levels of performance and energy efficiency, the flexible and scalable nature of Mali GPUs allows partners to design their SoC to their individual requirements. Partners Deepoon and Nibiru have recently launched awesome Mali-powered standalone VR devices for this very reason and the recently released Mali-G71 GPU takes this another step further. Not only does it double the number of available cores but it also provides 40% bandwidth savings and 20% more energy efficiency to allow SoC manufacturers to strike their ideal balance between power and efficiency.
How foveated rendering displays only the immediate field of view in high resolution
Another potentially game-changing use of eye tracking is for security and authentication. Retinal scanning is not an unfamiliar concept in high-end security systems so to extend the uses of eye tracking to this end is a logical step. The ability to read the user’s retinal ID for in-app purchases, secure log in and much more not only reduces boring verification steps but simultaneously makes devices and applications much more secure than they were before! So once you’ve used your unique retinal ID to access your virtual social media platform, it doesn’t stop there right? Of course not, social VR would be a weird place to hang out if your friends’ avatars never looked you in the eye. Eye tracking can take this kind of use case to a whole new level of realism and really start to provide an alternative way to catch up with people you maybe rarely get to see in person. Early examples are already inciting much excitement for the added realism of being able to interpret eye contact and body language.
Seemingly simple innovations like this can actually have a huge impact on an emerging technology like VR and provide incremental improvements to the level of quality we’re able to reach in a VR application. Foveated rendering in particular is a huge step up for bandwidth reduction in the mobile device so with advancements like these we’re getting ever closer to making VR truly mainstream.