Today’s Arm-powered smartphones are the hub of everything we do as people. From everyday tasks and productivity, such as communication, shopping and banking, through to more complicated workloads and experiences that are driving technology requirements, such as video streaming, gaming, virtual reality (VR) and augmented reality (AR). These experiences are what Arm refers to as digital immersion, a key theme from Arm’s Mobile IP launch in May 2020 and one that you’ll see again at Arm DevSummit. Alongside these experiences, 5G, Machine Learning (ML) and security are the enabling technologies that make digital immersion richer, more fulfilling and more secure for the end-user.
95 percent of the world’s shipped smartphones use Arm Cortex CPUs and 50 percent use Arm Mali GPUs, with Arm’s role crucial to creating the technology blueprint to deliver performance and efficiency on mobile devices. Laying the groundwork to enable our partners to provide innovative and safe digital immersion experiences on consumer devices.
During the unprecedented COVID-19 global health crisis, the role of technology in connecting, aiding, and supporting everyone has been more important than ever. Many of our lives during these challenging times have been lived through our smartphones. We have communicated with friends and family through video calling and various messaging apps. We have organized food shopping deliveries through the web browsing capabilities. We have organized financial matters through banking apps. Finally, our smartphones have helped to ease us into working from home through access to work emails and other productivity apps.
Alongside these everyday tasks, the range of digital immersion experiences are becoming increasingly complex. Mobile gaming is one use case that is expanding with more gaming apps and more complex gaming experiences. However, we need the appropriate technology to make this possible.
Mobile Gaming - an important digital immersion experience
A key role for Arm is supporting the gaming ecosystem – which includes device manufacturers, silicon vendors, OSVs, game engines and game developers – to enable gaming on mobile. We provide the IP, technologies, and tools that push the boundaries of graphics and optimize mobile gaming experiences.
The greater performance of the new Arm Cortex-A78 CPU and Arm Mali-G78 GPU, coupled with the fast speeds and high bandwidth of 5G (more on that later), enables premium gaming experiences on mobile. The maximum 24-core count coupled with Asynchronous Top Level on Mali-G78 enable our highest ever GPU performance in the most efficient and effective way possible. Moreover, the efficiency benefits of Cortex-A78 and Mali-G78 provide longer battery life on smartphones for more playing time without needing to charge.
Our ecosystem work further enhances performance to help build richer gaming experiences. Arm worked with Unity to bring the power of the Burst Compiler to Android, further enhancing multiprocessor performance and power management. We have also been providing faster mobile gaming experiences through updatable Mali GPU Drivers and the new Android GPU Inspector. Finally, we have worked closely with Crytek and Google to ensure that Crytek’s flagship ‘Neon Noir’ game demo fully utilizes Vulkan on Arm Mali to achieve outstanding graphic fidelity on mobile. These examples demonstrate the need to optimize our IP to continue to push the boundaries of performance for mobile gaming.
While gaming is well established in the mobile ecosystem, XR (which covers both AR and VR) is an emerging digital immersion experience. Like mobile gaming, Arm technology is powering the next-generation of XR-capable devices, from smartphones (where Arm technology is well established) to new wearable devices.
We are already seeing AR being used in the common daily activities of consumers, such as translation, navigation, instructions (for example, cooking and DIY) and gaming, on their smartphones. For AR translation, Google Translate scans text in a foreign language through the built-in camera of smartphones and translates it into text of the language that the user understands. Meanwhile for AR navigation the built-in cameras scans and identifies sites and landmarks before shifting through search results and providing the relevant information.
Unlike AR, VR is being mainly experienced on wearable devices, with gaming a huge market. We are already seeing many existing console and PC-based titles being adapted for VR, adding a new layer of immersion to the gaming experience. This is possible through the new wave of standalone VR devices, such as the Oculus Quest that is powered by Arm Cortex CPU technology.
The performance and power efficiency benefits of Arm technology mean users can enjoy immersive and interactive VR over a prolonged playing time. Our IP enables more and better pixels for longer and deeper experiences, while the area, bandwidth and efficiency savings support the development of lighter and more comfortable headsets for consumers.
As XR features continue to evolve and advance, more highly integrated compute solutions – which are based on Arm IP but delivered by our innovative partners – will be needed. These will deliver increased performance for complex XR workloads, alongside much-needed power-efficiency geared towards the lighter and smaller device form factors.
Another part of digital immersion is the enabling technologies that make the experiences possible on consumer devices. Three key enabling technologies worth highlighting are 5G, ML, and security.
The advent of 5G brings greater speed, connectivity, and reduced latency – especially for high-bandwidth applications. As a result of 5G, current digital immersion experiences on devices will substantially improve and, at the same time, completely new use cases will be created. For example, mobile cloud gaming.
5G-ready smartphones are set for huge growth in the next few years. In fact, an Arm-commissioned report by Newzoo on the impact of 5G on gaming predicts that over one billion 5G-ready smartphones will be in the market by the end of 2022. This is driving the need for higher-performing smartphones and more performant and efficient IP and solutions.
Machine Learning on the device - a key enabling technology
ML continues to transform all applications and devices, with this advancing throughout the last few years. Several years ago, we had smartphone functions that included face unlock and voice user interface. However, today ML-based experiences are expanding to deliver those that are ‘under-the-hood’ and visual. ML ‘under-the-hood’ is now responsible for device features such as power management. This predicts usage and adjusts device systems on and off to maximize battery life. Meanwhile, the visual ML-based experiences are higher-quality photographs and videos and brand-new AR-based applications.
Arm provides the IP and best-in-class software and tools needed to deploy the next-generation of intelligent devices. For endpoint ML, Arm high-performant IP enable the processing of ML workloads on devices rather than the cloud, easing privacy and security concerns for users. Arm CPUs support the most popular real-world applications and use-cases on smartphones, such as social media filters, dictation, translation, and security functions. However, the IP implementations for ML workloads vary based on the ML requirements of the device, the use case, and the specific workloads. Although the GPU’s primary function is graphics processing, its parallel data processing capability makes it suitable for running ML workloads linked to security features and different camera and video modes.
The final enabling technology is security. Consumer devices – particularly smartphones – are now a hub of valuable personal information and content, making them a high-value target for hackers. In fact, a 2019 survey by Arm and Northstar Research revealed that 35 percent of respondents would stop using their device if it was compromised and their personal data was stolen.
To fully enjoy digital immersion, consumers need to have trust in their devices. Arm has a rich history as a pioneer for security on mobile with TrustZone technology, which is used in billions of devices worldwide. The family of TrustZone technologies can be integrated into any Arm Cortex-A CPU core, supporting high-performance application processors.
The advances in digital immersion are creating more advanced, complex, and demanding multi-compute workloads. To realize these requirements, Arm believes that there needs to be a massive shift in the approach to SoC design. Total Compute from Arm moves beyond optimizing individual IP and takes a system-level solution view of the SoC, putting digital immersion use cases and experiences at the heart of the designs. This focuses on three key areas. Firstly, compute performance to deliver the complex digital immersion experiences of the future. Secondly, security to make sure these experiences are delivered securely and that consumer data is protected. Thirdly, developer access so developers can access this improved compute performance and use it to deliver improved user experiences through their applications.
Essentially, digital immersion describes all the ways that we, as consumers, interact with our devices. These devices and the experiences they provide are a fundamental part of everything we do now and will do in the future. As the number of consumer devices and quality and quantity of experiences that are possible on these devices continue to grow, so does the level of digital immersion and the impact this has on our daily lives.
Arm is committed to continuing the delivery of both performance and efficiency gains to ensure our partner ecosystem can offer greater digital immersion on the next generation of mobile devices. The Mobile IP launch in May 2020 represented the latest example of this commitment. However, the move towards greater digital immersion will also lead to more advanced, complex, and demanding multi-domain compute workloads. This is where a new Total Compute approach is needed. As the mobile future remains built on Arm technology, we are committed to enable even greater and more advanced digital immersion in the future.
Learn more about Arm's latest Mobile IP launch