IoT continues to transform edge computing across a huge number of markets and applications, from the smallest, power-constrained devices to embedded devices with displays, graphics, and compute capabilities. Arm is at the forefront of this transformation and continues to empower this transformation by providing Arm Virtual Hardware as part of Arm Total Solutions for IoT.
Arm Virtual Hardware is an evolution of Arm's modeling technology that provides functionally accurate models of Arm-based SoCs. It provides application developers the chance to build and test software before and after hardware availability. It runs as a simple application in the cloud for simulating memory and peripherals, for testing and use of modern agile software development practices such as DevOps and MLOps workflows.
Himax Technologies, Inc., a leading supplier, and fabless manufacturer of display drivers and other semiconductor products such as edge AI processors and always-on image sensors has been collaborating with Arm in this transformation. Himax always-on smart sensing products provide cutting edge AI computing capabilities along with extremely low power consumption, low-bandwidth requirement and privacy. To embrace the fast-growing market of ultra-low power technologies with intelligence on edge devices, Himax has been one of the leading partners of Arm Total Solutions by using Arm Virtual Hardware for its internal software development and machine learning efforts.
The convergence of machine learning and artificial intelligence has significantly improved the IoT’s potential and become a key part of the IoT transformation. As part of these efforts, Himax has also joined the Arm AI Partner Program to offer leading edge AI expertise, solutions, and services to developers and customers, enabling the transformation with Arm.
Himax has demonstrated the successful utilization of the Arm Virtual Hardware based on Corstone-300 and configured its projects to the target platforms. On the Arm Virtual Hardware, Himax was able to use an ultra-light weight machine learning model, YOLO fastest XL, a public model with the capability to detect around 80 objects, representing a perfect deployment of tinyML applications.
Hear from Mark Chen, VP R&D at Himax Technologies discussing how Himax are leveraging Arm Virtual Hardware.
To get started with Arm Virtual Hardware check out this documentation page, which provides instructions about how to access the Amazon Machine Image (AMI) from the AWS marketplace. The documentation explains how to launch an EC2 instance of Arm Virtual Hardware and connect.
A good first software example to try is the micro-speech example using TensorFlow Lite. It has a very simple procedure to compile and run.
git clone https://github.com/ARM-software/VHT-TFLmicrospeech.git
The micro-speech example prints the output from audio inference results of recognizing yes and no keywords.
The micro-speech example runs with the default configuration of the Corstone-300 system. Now that the Arm Virtual Hardware AMI is running and confirmed to execute the micro-speech, let us try the Arm ML embedded evaluation kit.
To enable Arm Virtual Hardware for the Himax specific configuration, the number of MACs should be set to 64. This reflects Himax's configuration and provide the best estimation of performance. Changing the number of MACs should be done both on the software compilation step and on the Arm Virtual Hardware execution step.
When connected to the Arm Virtual Hardware AMI, clone the example project.
git clone https://review.mlplatform.org/ml/ethos-u/ml-embedded-evaluation-kit
git submodule update --init
Full documentation is provided with the kit, but to get started, to build all the examples for 64 MACs, using the Arm Compiler for Embedded, please enter:
./build_default.py --npu-config-name ethos-u55-64 --toolchain arm
The build script passes the number of MACs to the Vela compiler using the --accelerator-config parameter. A cmake folder is created with the built images therein the bin sub-folder.
VHT-Corstone-300, -C cpu_core.ethosu.num_macs=64 -a <image.axf>
for example, -a ethos-u-img_class.axf for the image classification example
If the number of MACs used in the compilation does not match the model configuration at runtime, the inference fails with an NPU config mismatch error. It is essential to check that the number of MACs is the same for the build and for the run.
Since Arm is enabling the Arm Virtual Hardware targets on an AWS AMI, it is easy for developers to kickstart their embedded software development. Developers can use either the default configuration or customers-specific configuration such as Himax to have both the embedded software and machine learning software ready before silicon is available. This reduces the overall time to market for IoT devices.
We expect more Arm customers and Himax's customers to be able to run more complex vision and voice-related use-cases on the Arm Virtual Hardware and to expedite their software and machine learning development to reduce their overall time to market.
This is the beginning of the new IoT evolution to further accelerate the adoption of IoT devices and reach 1 trillion devices by the year 2030.
Arm Virtual Hardware is available as a public beta on AWS Marketplace for Arm Corstone and Arm Cortex-M processors.