Arm Community
Arm Community
  • Site
  • User
  • Site
  • Search
  • User
  • Groups
    • Research Collaboration and Enablement
    • DesignStart
    • Education Hub
    • Innovation
    • Open Source Software and Platforms
  • Forums
    • AI and ML forum
    • Architectures and Processors forum
    • Arm Development Platforms forum
    • Arm Development Studio forum
    • Arm Virtual Hardware forum
    • Automotive forum
    • Compilers and Libraries forum
    • Graphics, Gaming, and VR forum
    • High Performance Computing (HPC) forum
    • Infrastructure Solutions forum
    • Internet of Things (IoT) forum
    • Keil forum
    • Morello Forum
    • Operating Systems forum
    • SoC Design and Simulation forum
    • 中文社区论区
  • Blogs
    • AI and ML blog
    • Announcements
    • Architectures and Processors blog
    • Automotive blog
    • Graphics, Gaming, and VR blog
    • High Performance Computing (HPC) blog
    • Infrastructure Solutions blog
    • Innovation blog
    • Internet of Things (IoT) blog
    • Operating Systems blog
    • Research Articles
    • SoC Design and Simulation blog
    • Tools, Software and IDEs blog
    • 中文社区博客
  • Support
    • Arm Support Services
    • Documentation
    • Downloads
    • Training
    • Arm Approved program
    • Arm Design Reviews
  • Community Help
  • More
  • Cancel
Research Collaboration and Enablement
Research Collaboration and Enablement
Research Articles Neural network architectures for deploying TinyML applications on commodity microcontrollers
  • Research Articles
  • Arm Research - Most active
  • Resources
  • Arm Research Events
  • Members
  • Mentions
  • Sub-Groups
  • Tags
  • Jump...
  • Cancel
Research Collaboration and Enablement requires membership for participation - click to join
More blogs in Research Collaboration and Enablement
  • Research Articles

Tags
  • Arm Research
  • Machine Learning (ML)
Actions
  • RSS
  • More
  • Cancel
Related blog posts
Related forum threads

Neural network architectures for deploying TinyML applications on commodity microcontrollers

Colby Banbury
Colby Banbury
June 29, 2021
7 minute read time.

What is TinyML?

Data transmission tends to dominate the energy consumption of an Internet of Things (IoT) device [1], yet a significant portion of the data transmitted is not used. The reality is that the vast majority of data is not that interesting. However, we have to keep looking so we do not miss it when something significant does happens [2]. If IoT systems could be smart enough to only transmit the interesting data, they could last far longer on battery power and be less likely to flood the network with uninteresting data, resulting in efficiency gains. Enter ‘TinyML’, which seeks to deploy Machine Learning (ML) algorithms on ultra low power systems, to enable us to intelligently select which data to transmit, improving energy efficiency.

Challenges of TinyML

Deploying ML inference tasks on TinyML devices comes with a unique set of challenges. Arm Cortex-M class microcontrollers (MCUs) often have severely limited Flash and static random-access memory (SRAM) to store model weights and activations. Even size efficient models designed for mobile devices, like MobileNets [3], require an order of magnitude more memory than is typically available on MCUs. Furthermore, IoT applications often have to keep pace with the rate at which the data is collected, despite MCUs having relatively low computational resources. Finally, the faster we run the model, the more time we can spend in ‘sleep mode’. This is where we shut down the MCU to preserve energy and extend the battery life. To effectively perform ML on MCUs, we need to design optimal neural network architectures that fit the constraints of SRAM memory, Flash memory, and latency. MicroNets [4], our recent work published at MLSys 2021, tackles this challenge via neural architecture search.

Differentiable neural architecture search

We use differentiable neural architecture search (DNAS) to discover accurate neural network architectures while satisfying SRAM, Flash, and latency constraints. DNAS uses gradient descent to rapidly produce optimized models, but prefers constraints to be closed form functions. To accurately model our constraints, we performed an in-depth characterization of neural network performance on MCUs.

Hardware characterization 

To characterize the performance of TinyML models, we tested hundreds of models and related their on-device metrics to their architecture. We deployed the neural networks to an STM32f746ZG MCU using the Tensorflow Lite for Microcontrollers (TFLite Micro) inference framework.

SRAM and Flash memory

We determined that the SRAM consumption of the model is dominated by the size of the buffer for intermediate tensors used during neural network execution, with some additional capacity consumed by persistent buffers that must be maintained between inferences. These factors can be determined directly from the architecture of the model. The SRAM overhead of the TFLite Micro framework and the bare-metal OS on top of this is typically minimal.

The Flash consumption is a similar story. Where the weights and biases of the model account for, on average, the majority of the Flash consumption with a significant portion of the overall consumption coming from the quantization parameters. Both of these can be determined from the model architecture. TFLite Micro and the system code are, again, overheads that can be accounted for statically.

Breakdown of SRAM and Flash memory usage of an example Keyword Spotting model.Figure 1: Breakdown of SRAM and Flash memory usage of an example Keyword Spotting model.

Latency

We observed an interesting relationship between operation (Op) count and latency. Models with significantly different architectures can differ in Op throughput, which means actual on-device latency is the only way to accurately compare two distinct models. However, despite this, we actually observed that models sampled from the same backbone have very similar Op throughputs. Therefore, we are able to use model op count to predict latency, as long as we sample from the same backbone like we already do with DNAS. Op count is simple to calculate from the model architecture, and therefore, as with SRAM and Flash usage, we can predict it during the architecture search.

The latency of models sampled from two different backbones, measured on two MCUs.Figure 2: The latency of models sampled from two different backbones, measured on two MCUs.

MLPerf Tiny use cases

We target three use cases from MLPerf Tiny, an MLCommons benchmark for TinyML systems: visual wake words [5], keyword spotting [6], and anomaly detection [7].

Visual wake words is a binary image classification dataset where the task is to determine if a person is in the image or not. An example of this use case would be a smart doorbell, where the camera can notify you when a person is at the door. 

Keyword spotting is widely used in commercial IoT devices with households around the world using “Alexa” or “ok Google” to wake their devices multiple times a day. The dataset that we used has 12 labels, including 10 words to identify, as well as silence and unknown categories.

Anomaly detection is an unsupervised audio task where the goal is to identify faults or abnormal behavior in a machine through the sounds it produces. This use case has wide applicability in industrial settings, where an inexpensive microcontroller can be used to automatically detect faults in machinery for predictive maintenance.

Results

Using DNAS we produced a set of MicroNets which achieve state-of-the-art performance in all metrics on all three target use cases. Our models trade off accuracy and size to best meet the requirements of any application. All of our models achieve real-time performance on their target MCU. The results are shown below and compared against a number of baseline models.

The models are open source and available here.

Keyword Spotting MicroNets Results.Figure 3: Keyword Spotting MicroNets Results

Visual Wake Words MicroNets Results.Figure 4: Visual Wake Words MicroNets Results. TFLM refers to the stock example model provided by Tensorflow Lite for Microcontrollers.

Anomaly Detection MicroNets Results.Figure 5: Anomaly Detection MicroNets Results

Conclusion

TinyML has the potential to revolutionize IoT and democratize AI, but the hardware constraints of microcontrollers make it difficult to deploy accurate models. The Arm ML Research Lab has been working on this topic for a number of years, to develop compact and accurate models that run efficiently on MCUs [8][9][10] and also to enable distributed learning [11]. If you’re interested in these topics, please get in touch!  

MicroNets demonstrates that differentiable neural architecture search can be used to rapidly find state-of-the-art models that can be easily deployed to commodity MCUs. This technique can be extended to other TinyML use cases and enable more efficient deployment of IoT applications. We plan to conduct future research on efficient TinyML model and system design.

The MicroNets paper was published at MLSys 2021 and is available here. You can watch the presentation from the conference here.

Questions? Contact Paul Whatmough 

Want to reference the work? 

“MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers”, C. Banbury, C. Zhou, I. Fedorov, R. M. Navarro, U. Thakker, D. Gope, V. J. Reddi, M. Mattina, P. Whatmough, MLSys, 2021.

References

[1] Bouguera, Taoufik et al. “Energy Consumption Model for Sensor Nodes Based on LoRa and LoRaWAN.” Sensors (Basel, Switzerland) vol. 18,7 2104. 30 Jun. 2018, doi:10.3390/s18072104

[2] Mckinsey Global Institute. “The Internet of Things: Mapping the Value Beyond the Hype.” mckinsey.com 

[3] Howard, Andrew G., et al. "Mobilenets: Efficient convolutional neural networks for mobile vision applications." arXiv preprint arXiv:1704.04861, 2017.

[4] Banbury, Colby, et al. "Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers." Proceedings of Machine Learning and Systems 3, 2021.

[5] Chowdhery, A., Warden, P., Shlens, J., Howard, A., and Rhodes, R. Visual wake words dataset. arXiv preprint arXiv:1906.05721, 2019.

[6] Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.

[7] Purohit, H., Tanabe, R., Ichige, K., Endo, T., Nikaido, Y., Suefusa, K., and Kawaguchi, Y. Mimii dataset: Sound dataset for malfunctioning industrial machine investigation and inspection. arXiv preprint arXiv:1909.09347, 2019

[8]I. Fedorov, R. P. Adams, M. Matting, P. N. Whatmough, ”SpArSe: Sparse Architecture Search for CNNs on Resource Constrained Microcontrollers”, Advances in Neural Information Processing Systems (NeurIPS), 2019

[9] I. Fedorov, M. Stamenovic, C. Jensen, L.-C. Yang, A. Mandell, Y. Gan, M. Mattina, P. N. Whatmough, “TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids”, InterSpeech, 2020

[10] U. Thakker, P. N. Whatmough, Z. Liu, M. Mattina, J. Beu, “Doping: A technique for Extreme Compression of LSTM Models using Sparse Structured Additive Matrices”, Machine Learning and Systems (MLSys), 2021

[11] D. A. E. Acar, Y. Zhao, R. M.  Navarro, M. Mattina, P. N. Whatmough, V. Saligrama, “Federated learning based on dynamic regularization”, International Conference on Learning Representations (ICLR), 2021

Anonymous
  • Ankit Tiwari
    Offline Ankit Tiwari over 1 year ago

    Hi
    We are currently living in a world surrounded by Machine Learning models.
    TinyML is a field of study in Machine Learning and Embedded Systems that explores the types of models you can run on small,
    low-powered devices like microcontrollers. It enables low-latency, low power, and low bandwidth model inference at edge devices. 
    Hope this will help. 

    • Cancel
    • Up 0 Down
    • Reply
    • More
    • Cancel
Research Articles
  • Overcoming resistance

    Andrew Pickard
    Andrew Pickard
    Improving the characteristics of interconnects as device dimensions scale smaller.
    • September 22, 2022
  • Hands-on experience at Singapore Management University

    Andrew Pickard
    Andrew Pickard
    SMU has been working on the SAP Next-Gen student project, to develop innovative sustainability solutions using SAP software and real-world IoT devices from Arm's partner ecosystem.
    • May 30, 2022
  • Cryptography: what is under the mask?

    Andrew Pickard
    Andrew Pickard
    Sorbonne Université has been using Arm processor source code for modelling and verification on the hardware at the micro-architectural level.
    • May 26, 2022