Ray tracing is a computer graphics technique that generates realistic lighting and shadows by modelling the paths that individual light rays take around a scene. The technique has existed for decades and generates highly realistic results, but it was only in 2018 that the introduction of hardware accelerated ray tracing for desktop PCs meant that real-time ray traced experiences in games became viable. The inclusion of hardware acceleration in the latest generation of games consoles has further motivated studios to take advantage of ray tracing techniques in their games. This produces the impressive visuals seen in Fortnite, Minecraft, Cyberpunk 2077, Spiderman, Ratchet and Clank Rift Apart and many others.
According to Newzoo, high-fidelity mobile gaming is rapidly growing across the globe. In China, the revenue share of high-fidelity mobile games grew from 42% in 2016 to 70% in 2020, and in North America it grew from 6% in 2016 to 33% in 2020. With this growing appetite for complex gaming, developers are already starting to think how to cater to the needs of their audience.
But nothing comes for free. Even when hardware is accelerated, a developer using ray tracing must carefully balance visual quality with the resulting frame rate to create a compelling experience for their players, and even though ray tracing techniques are more efficient than ever, they can still cost significant energy.
Understanding this balance between quality, FPS and power is vital as we start to explore the application of ray tracing techniques to mobile device content. This is because whatever makes sense for large-screened, power-rich desktop and console environments will not work for the mobile market, where screens are small (but very high density) and maximizing battery life is the key to retaining players. We also need to consider the type of gaming that takes place across the diverse range of mobile devices in the market; ray tracing might deliver some great eye-candy. But, what if realistic reflections and shadows gave you an edge in the competitive multiplayer games that are pushing the limits of mobile devices? Would such a thing be welcomed by the communities that mobile developers serve?
These are questions that need to be answered, and we can start now! We are working to enable raytracing for Arm Mali – the world’s most popular mobile GPU, including the billions of devices that are already in the hands of gamers today. Furthermore, at the ‘Future of the Arm Architecture’ talk at Arm Vision Day, we announced that future Mali GPUs will provide hardware acceleration that significantly increases the efficiency of ray tracing techniques on mobile devices. This is on top of our recently announced latest generation of Mali GPUs, including Arm Mali-G710, which brings a 20% boost to performance and efficiency compared to Arm Mali-G78.
In this blog, we explain how we have been developing mobile ray traced content, and how we are working with partners like Tencent and MediaTek to make it easier for developers to explore the benefits that ray tracing could bring to the world of mobile gaming.
The Arm technology that powers the world’s smartphones is licensed, configured and augmented by our partners to produce an extraordinarily diverse range of devices that span all geographies, price points and form factors. The $90.7Bn mobile gaming market is made possible by 2.8 billion mobile gamers, and studios know that maximising revenues means maximising their addressable market. For example, the fantastic success of Garena Free Fire is due, in part, to the game’s small memory footprint and download size, allowing them to address 97.4% of the market and capture more than 666 million users in 2020 as a result. Our partner King also noted that generating mass market appeal through scalability and optimizations were also vital in them bringing one of console gaming’s most recognized franchises, Crash Bandicoot: On the Run! to mobile. This is further reinforced by Newzoo’s latest report, which outlines how games’ specs directly impacts their uptake and addressable market.
Standardization across devices is critical to making the life of a games developer more manageable. The release of the Vulkan Ray Tracing extensions in November 2020 made this standardization a reality for ray tracing, with an API that Arm and our partners will align on to ensure that ray traced content is functionally portable across our ecosystem’s huge range of devices.
Hardware architects have a responsibility to teach the software ecosystem how to get the best out of mobile devices, and it is a responsibility that we at Arm take very seriously. We have donated our Vulkan Samples to Khronos, developed free performance analysis tools for Android and released extensive education for graphics developers. We are leading mobile ray tracing and preparing the ecosystem through our recent partnerships with MediaTek and Tencent.
We are currently developing a software implementation of the Vulkan Ray Tracing API, specifically the Ray Query element, which provides the easiest way for mobile developers to start adding ray tracing effects to their games. The implementation has been developed as a Vulkan Layer, which can be loaded at runtime by the developer to seamlessly add ray tracing capabilities to the graphics driver on their chosen platform.
The result of this can be seen in our in-house developer demo called Bonza:
In the video, Bonza is running on a device with Mali-G78, and ray tracing is used to generate all shadows. In this indoor scene, you can see two light sources: the sun from the outdoors (you can see the shadows from the windows pushing across the walkway) and an indoor point light to help accentuate the shadow of the focal Bonza statue. Drones move through the scene, teaching us how to update the acceleration structures that the Vulkan Ray Query API uses to determine what a given light ray will hit. This is important because modern game environments are far from static, and so any practical use of ray tracing will need to work well with the advanced physics-based destructible terrain featured in games, such as Battlefield V.
Beyond our work on dynamic shadows, Tencent have been looking at efficient ways to generate soft shadows, as well as using ray query to improve the quality of ambient occlusion. This is a graphics technique that adds realism to a scene by approximating the amount of geometry around a particular point, and thus the amount of ambient light that will fall on that it.
"Think of how illumination plays an integral role for experiencing the real world with human vision, real-time ray tracing is reasonably believed to be the crown jewel of computer graphics in our generation. With the tremendous effort by our engineers and artists, and the strong support from MediaTek and Arm, we are ready to begin the next chapter of high-fidelity mobile gaming empowered by the revolutionary ray tracing technologies", said Nan Wei, Deputy Director of Engine Technology at Tencent Games.
You can read more about Tencent’s approach to mobile ray tracing in their recent blog.
Although still work in progress, these demos have already taught us a lot about how to create optimized ray traced content for mobile devices. Irrespective of how efficiently your hardware can process ray queries, you always want to:
For more technical detail about what we have learned so far, check out our talk that Arm, Tencent and MediaTek presented at GDC 2021 in July 2021. In this talk, we describe our work on the software emulation layer and the best practices for mobile ray tracing that we are developing.
To further help the ecosystem and this technology’s adoption, MediaTek has introduced its MediaTek RTSDK for developers. Its powerful Dimensity 5G chipsets are already featured in millions of smartphones worldwide, giving developers a large userbase ready to experience new in-game visuals in lighting, reflection and refraction effects. The MediaTek RTSDK is fully compatible with the Khronos standards, allowing PC developers to migrate smoothly to mobile platforms. It provides an avenue to create core ray tracing functions, including path tracing, screen-space triggered ray-traced shadows (avoiding shadow-map aliasing issues), multiple bounced refraction (avoiding object sorting), and arbitrary surface reflection.
"The collaboration with Arm and Tencent has enabled us to bring advanced graphics capabilities, such as Ray Tracing, to our Dimensity 5G chips. Using our new RT SDK solutions, game developers can now easily update their games with new capabilities, enhancing the in-game experience", said Dr. Eddie Tsao, Assistant General Manager of Computing and AI Technology Group at MediaTek. “Since Dimensity chips are already found in millions of global smartphones, we are uniquely enabling the most vivid visuals to the widest audience possible.”
We will also be talking more about Ray Tracing at Arm DevSummit, which runs online from the 19th to the 21st October. DevSummit is free to attend and includes a great line-up of technical talks – plenty for graphics and mobile developers, as well as covering cloud, automotive, IoT, AI, HPC and hardware design. You can register here, with early registration allowing you to watch technical sessions from 6th October, access our Discord server and secure you space at the virtual campfires and workshops.
We are always keen to collaborate with more partners who share our interests, so if you’re working on mobile ray tracing and have some best practices to share, or would like to work with us on building out our portfolio of ray tracing content and demos, please get in touch at developer@arm.com.