AR and ML using Google ARCore, Arm NN in Unity running on mobile devices

// Helper function to make input tensors
armnn::InputTensors MakeInputTensors(const std::pair<armnn::LayerBindingId,
 armnn::TensorInfo>& input,
 const void* inputTensorData)
{
 return { { input.first, armnn::ConstTensor(input.second, inputTensorData) } };
}
 
// Helper function to make output tensors
armnn::OutputTensors MakeOutputTensors(const std::pair<armnn::LayerBindingId,
 armnn::TensorInfo>& output,
 void* outputTensorData)
{
 return { { output.first, armnn::Tensor(output.second, outputTensorData) } };
}
 
extern "C" __attribute__ ((visibility ("default")))
int detectObjects(float *inputPtr, float *result)
{
 float *outputPtr = s_OutputBuffer;
 armnn::Status ret = s_Runtime->EnqueueWorkload(s_NetworkIdentifier,
 MakeInputTensors(s_InputBindingInfo, inputPtr),
 MakeOutputTensors(s_OutputBindingInfo, outputPtr));
 if (ret == armnn::Status::Failure)
 {
 throw armnn::Exception("IRuntime::EnqueueWorkload failed");
 }
 

You may need to implement the ParseOutputTensorsYoloV1 yourself. GitHub has some useful code snippets that may help you to implement the YOLO v1 parser.

Use the NDK standalone toolchain to compile the above code, and then generate the "libyoloDetector.so" shared library. In order to call from Unity, you should create a folder named "Assets > Plugins > Android" and copy the armeabi-v7a shared libraries to that folder in your Unity project. Here are the libraries I copied:

• libarmnn.so
• libarmnnCaffeParser.so
• libprotobuf.so
• libc++_shared.so
• libyoloDetector.so

Integrate Arm NN into Unity

Let's switch back to the Unity project we created. We need camera raw data to feed into the object detection model, but the Google ARCore already took control of the camera. Fortunately, Google ARCore realized other programs may want to access the camera raw data as well. Thus, they have provided an example for this. You can see the "Assets > GoogleARCore > Examples > ComputerVision" example for more detail. We could use the TextureReader script to do the same thing in our example.

1. Select the Main scene in the Hierarchy tab.
2. Select "GameObject > Create Empty" to create an empty game object.
3. Rename the empty game object to "ArmNNCaffeParserController"
4. Click the "Add Component" button in the inspector.
5. Search "Texture Reader" and add it to the "ArmNNCaffeParserController".
6. Change the Image Width and Height to 448.
7. Change "Image Sample Mode" to "Keep Aspect Ratio"
8. Change "Image Format" to "Image Format Color"

You should get this in the end.

Create two C Sharp scripts for this demo.

1. Create a "Assets > Scripts" directory.
2. Create two C Sharp scripts named "ArmNNCaffeParserController.cs" and "ArmNNCaffeDetector.cs".
3. Select "ArmNNCaffeParserController" in the "Main" scene.
4. Use "Add Component", search for "Arm NN Caffe Parser Controller" and add it to the controller.

In the "ArmNNCaffeDetector.cs" script, call the initDetector native function in the constructor.

private static int INPUT_SIZE = 448;
private static int RESULT_SIZE = 7 * 7 * 6;
private float[] fetchResults = new float[RESULT_SIZE];
 
[DllImport ("yoloDetector")]
private static extern void initDetector();
 
public ArmNNCaffeDetector()
{
 inPtr = Marshal.AllocHGlobal(3 * INPUT_SIZE * INPUT_SIZE * sizeof(float));
 outPtr = Marshal.AllocHGlobal(RESULT_SIZE * sizeof(float));
 initDetector();
}

And call the detectObjects native function in the newly created DetectAsync method.

[DllImport ("yoloDetector")]
private static extern int detectObjects(IntPtr input, IntPtr output);
 
public Task<List<KeyValuePair<DetectResults, float>>> DetectAsync(byte[] camImage)
{
 return Task.Run(() =>
 {
 // Prepare input here
 ...
 Marshal.Copy(inputBuffer, 0, inPtr, inputBuffer.Length);
 int detectObjectNums = detectObjects(inPtr, outPtr);
 Marshal.Copy(outPtr, fetchResults, 0, RESULT_SIZE);
 ...
 // Parse and return the results here
 });
}

>> DetectAsync(byte[] camImage) { return Task.Run(() => { // Prepare input here ... Marshal.Copy(inputBuffer, 0, inPtr, inputBuffer.Length); int detectObjectNums = detectObjects(inPtr, outPtr); Marshal.Copy(outPtr, fetchResults, 0, RESULT_SIZE); ... // Parse and return the results here }); }]]>

Before the detectObjects call, you may need to convert the camera data format to ArmNN order. Here is the code snippet to do that.

float[] inputBuffer = new float[INPUT_SIZE * INPUT_SIZE * 3];
int h = INPUT_SIZE;
int w = INPUT_SIZE;
int c = 4;
for (int j = 0; j < h; ++j)
{
 for (int i = 0; i < w; ++i)
 {
 int r, g, b;
 r = camImage[j * w * c + i * c + 0];
 g = camImage[j * w * c + i * c + 1];
 b = camImage[j * w * c + i * c + 2];
 
 // ArmNN order: C, H, W
 int rDstIndex = 0 * h * w + j * w + i;
 int gDstIndex = 1 * h * w + j * w + i;
 int bDstIndex = 2 * h * w + j * w + i;
 
 inputBuffer[rDstIndex] = (float)r/255.0f;
 inputBuffer[gDstIndex] = (float)g/255.0f;
 inputBuffer[bDstIndex] = (float)b/255.0f;
 }
}

In the "ArmNNCaffeParserController.cs" script, instantiate the ArmNNCaffeDetector class and setup a callback function for TextureReader.

TextureReader TextureReaderComponent;
private ArmNNCaffeDetector detector;
 
private int m_ImageWidth = 0;
private int m_ImageHeight = 0;
private byte[] m_CamImage = null;
 
private bool m_IsDetecting = false;
 
void Start () {
 this.detector = new ArmNNCaffeDetector();
 
 TextureReaderComponent = GetComponent<TextureReader> ();
 
 // Registers the TextureReader callback.
 TextureReaderComponent.OnImageAvailableCallback += OnImageAvailable;
 Screen.sleepTimeout = SleepTimeout.NeverSleep;
}

(); // Registers the TextureReader callback. TextureReaderComponent.OnImageAvailableCallback += OnImageAvailable; Screen.sleepTimeout = SleepTimeout.NeverSleep; }]]>

Implement the OnImageAvailable to receive the camera data and call the ArmNNDetect method.

public void OnImageAvailable(TextureReaderApi.ImageFormatType format, int width, int height, IntPtr pixelBuffer, int bufferSize)
{
 if (format != TextureReaderApi.ImageFormatType.ImageFormatColor)
 {
 Debug.Log("No object detected due to incorrect image format.");
 return;
 }
 
 if (m_IsDetecting) {
 return;
 }
 
 if (m_CamImage == null || m_ImageWidth != width || m_ImageHeight != height)
 {
 m_CamImage = new byte[width * height * 4];
 m_ImageWidth = width;
 m_ImageHeight = height;
 }
 System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, m_CamImage, 0, bufferSize);
 
 m_IsDetecting = true;
 Invoke(nameof(ArmNNDetect), 0f);
}

Call the DetectAsync method in the ArmNNDetect method.

private async void ArmNNDetect()
{
 var probabilities_and_bouding_boxes = await this.detector.DetectAsync (m_CamImage);
 ...
 // Visualize the bounding boxes and probabilities to the screen
 // Use "Frame.Raycast" which ARCore provided to find the 3D pose of the detected objects.
 // And render a related virtual object at the pose.
}

The DetectAsync method will return the probabilities and bounding boxes for the detected objects. After that, do whatever you want to do there. e.g. visualize the bounding boxes and push some virtual content nearby the physical objects.

How to use the "Frame.Raycast" function to get the 3D pose of the detected objects? Do you remember the code you comment out in the Update method of the "HelloARController.cs" script? You can refer to that code, and use the bounding box coordinate instead of the touch point coordinate.

// Raycast against the location the object detected to search for planes.
TrackableHit hit;
TrackableHitFlags raycastFilter = TrackableHitFlags.PlaneWithinPolygon |
 TrackableHitFlags.FeaturePointWithSurfaceNormal;
 
if (Frame.Raycast(boundingbox.position.x, boundingbox.position.y, raycastFilter, out hit))
{
 var andyObject = Instantiate(AndyAndroidPrefab, hit.Pose.position, hit.Pose.rotation);
 
 // Create an anchor to allow ARCore to track the hitpoint as understanding of the physical
 // world evolves.
 var anchor = hit.Trackable.CreateAnchor(hit.Pose);
 
 // Andy should look at the camera but still be flush with the plane.
 if ((hit.Flags & TrackableHitFlags.PlaneWithinPolygon) != TrackableHitFlags.None)
 {
 // Get the camera position and match the y-component with the hit position.
 Vector3 cameraPositionSameY = FirstPersonCamera.transform.position;
 cameraPositionSameY.y = hit.Pose.position.y;
 
 // Have Andy look toward the camera respecting his "up" perspective, which may be from ceiling.
 andyObject.transform.LookAt(cameraPositionSameY, andyObject.transform.up);
 }
 
 // Make Andy model a child of the anchor.
 andyObject.transform.parent = anchor.transform;
}

There we have it. You should have your own AR/ML demo up and running! Did you do something differently? Why not share it with us in the comments?

Arm NN SDK is a free of charge set of open-source Linux software and tools that enables machine learning workloads on power-efficient devices. It provides a bridge between existing neural network frameworks and power-efficient Arm Cortex CPUs, Arm Mali GPUs or the Arm Machine Learning processor.

[CTAToken URL = "https://developer.arm.com/products/processors/machine-learning/arm-nn" target="_blank" text="Learn more about Arm NN SDK" class ="green"]