Earlier this year Arm collaborated with my company SensoMotoric Instruments to enable foveated rendering on a mobile virtual reality (VR) platform. The device was demoed at GDC'17 in California, a joint-showcase that proved to be a great success.
This was a big deal because, while the VR community agrees foveated rendering to be crucial – and by now inevitable - in tethered VR headsets, delivering it in a mobile device is more of a challenge - but it also brings greater benefits.
Foveated rendering allows us to use our knowledge of human perception to save significant amounts of computational power. The point at which the human eye is unable to distinguish reality in a VR display involves a resolution of about 16k per eye.
But when we look at a fully rendered scene in VR, much of the computational power is wasted because our eyes can only take in fine details at the very center of our vision. By understanding exactly where the user is looking, we can tell the processor to render only this area with full resolution. That’s foveated rendering.
And while that’s the basic theory of foveated rendering, achieving it isn’t that simple. Our eyes move rapidly (up to 1000 deg/sec), and this calls for technology that interprets the user’s point of gaze quickly and efficiently. SMI employs fast, accurate and low latency eye tracking which provides the user’s gaze coordinates almost instantaneously to the rendering pipeline. This means the center of attention always matches the full resolution area.
You know when you have been successful with foveated rendering when it is imperceptible. If the user can see it, it hasn’t worked. The benefits to VR in general are by now, obvious, and to mobile VR, there are even greater dividends. Apart from the reduced processing load on your mobile phone, there is the fact that foveated rendering reduces heat issues, lowers power demand and in turn, improves battery life.
SMI’s work with Arm showed what is possible in this area, proving to be a timely and important glimpse into the future of mobile VR.