MWC18: A Smartphone Camera With a Brain

Mobile continues to be a very dynamic industry, thriving with technical innovation, particularly around camera related technologies. In 2017 over 1.2 Trillion (that is 1,200,000,000,000) photos were taken with mobile devices. With over 1.5bn devices shipping every year, the smartphone is not only the obvious platform for driving technological innovation and the main vehicle for monetization, but it is the platform that can reach the most people around the globe.

We are very excited to be able to announce some of our recent collaborations in this area, some of which we will be showcased at the Mobile World Congress in Barcelona.

Working with Facebook to optimize machine learning for mobile

Facebook enables developers to create innovative mobile experiences that connect people around the world, with advanced technologies like the Camera Effects Platform introduced at the Facebook F8 conference last year.

These features use Facebook's machine learning technology to enhance the intelligence and performance of the mobile user's camera, and Arm has teamed up with Facebook to optimize the Caffe2 machine learning framework for the Arm Mali Graphics Processing Unit (GPU).

Throughout 2017, the two companies collaborated to optimize key machine learning neural networks for the Arm Mali GPU, and we are pleased to announce that this code is now freely available to developers. The code can be sourced directly from the Compute Library or used in conjunction with the Caffe2 framework.

Caffe2 is a lightweight and scalable framework for the development and deployment of machine learning models. Used across data centers to mobile and embedded devices, the modularity and flexibility of Caffe2 makes it a perfect fit for machine learning developers.

As a result of the collaboration between Arm and Facebook, Caffe 2 developers can now benefit from performance improvement on mobile devices using Arm GPUs. The collaboration will also extend to future Arm ML processors via the Compute Library’s existing APIs.

The Caffe2 framework is compatible with the new Arm ML platform and can take advantage of current and future Arm ML processors via the Compute Library's existing APIs as well as the newly announced Arm NN software.

In addition, Arm’s commitment to open standards was the reason we joined the ONNX community project initiated by Amazon, Facebook, and Microsoft to enable interoperability between AI network models (http://onnx.ai). You can read more about that project here.

A more intelligent and power efficient camera

For many years Arm has been collaborating closely with ArcSoft to optimize camera technologies for several mobile OEMs. At the Mobile World Congress this year we are demonstrating the ArcSoft PicSelfie technology, which has been accelerated using the new Google Android Neural Network API.

Here's a guide to how the application works:

• The user will point the smartphone camera as usual, in order to take a photo; the subject may be a person, a group, an object, or even a landscape
• The ArcSoft PicSelfie engine intelligently distinguishes the subject of the photo and what constitutes the background, then applies different processing techniques to each item
• For the background, the algorithm will apply a Bokeh effect to blur it and in turn make the subject more prominent
• For the primary subject of the photo the algorithm automatically applies beautification

All of this works happens in real time and is intelligently and dynamically done by the software. The technology works with both single and dual-lens devices, however with the latter, performance and quality is improved as depth is calculated more accurately.

ArcSoft PicSelfie - Dual-lens version of the algorithm

This technology is unique because the application uses the new Google Android Neural Network API (NNAPI), which was introduced last year and is supported on all devices running Android 8.1 (API level 27) or higher. This will work on every device with Arm Cortex CPUs using Arm NEON acceleration. However, through the use of Arm NN and the Compute Library mentioned above, the application is able to make use of additional hardware, such as a Mali GPU, to achieve even higher performance and responsiveness.

On a device with a single camera (no depth information) the algorithm uses traditional segmentation algorithms to separate the subject from the background. The Mali GPU is used to accelerate the segmentation phase via Google Android NNAPI, which already takes advantage of the Arm NN software. You can read more about Arm NN here.

Here is an overview of the processing stages for a single camera device, where segmentation is offloaded to the GPU using Arm NN:

ArcSoft PicSelfie - Single-lens version of the algorithm (block diagram)

If the device sports a stereo camera, the operation is slightly different in that depth information is used to help separate the subject from the background. The depth-map calculation is offloaded to Google Android NNAPI which then takes advantage of the Arm NN software to perform the operations on the GPU.

ArcSoft PicSelfie - Dual-lens version of the algorithm (block diagram)

Initial analysis by ArcSoft has shown a typical performance improvement of 30% whilst reducing power consumption by 20% when using the GPU to accelerate the ML workload.

For more details about Arm support for Google Android NNAPI see the 'Arm support for Android NNAPI gives >4x performance boost' blog post on the Arm Community.

To find out more about Arm new ML compute platform I recommend:

• The announcement of Project Trillium: 'Arm’s Project Trillium Offers the Industry's Most Scalable, Versatile ML Compute Platform' annoucement on the Arm website
• This blog by Jem Davies: 'Arm is changing machine learning experiences: Project Trillium' blog post on the Arm Community