As someone who has been building embedded Linux systems for the past 16 years, I am proud of the amazing products built with Linux on Arm. Yet, I am also concerned about how much effort is required to bring up every platform. By any measure, the embedded Linux ecosystem is a healthy one. It is used in every industry. The flexibility and functionality of Linux is easy to adapt into new products, and the open-source ecosystem enables all of us to collaborate and make Linux even better.
On the other hand, the software stack on most embedded devices is entirely custom. The OS is tied to the platform, and even the smallest change means rebuilding the entire image. For example, over the lifetime of a product the hardware design may change several times to account for component availability, to improve reliability, or for cost reduction. While outward functionality of the product may not change, each iteration may require a new OS image which supports the vendor long term. Similarly, many products use the same base hardware design and only differ on the user-space application software. However, in typical embedded development each variant has a different OS image that is supported separately and without any shared binary components.
The problem here is not building embedded systems. The excellent tools we have in Yocto, Buildroot, OpenWRT, and other make it easy to build an embedded product. Rather, the problem is that each build configuration requires extra effort and cost to support and maintain. There is not a good way to amortize support across multiple products by using the same binary components. An example is security patches: with one or a few build configurations you may be able to keep up to date with security patches, but the effort required scales linearly with every additional build. If you have 100 products, then every time a security patch is applied, 100 different kernel configurations need to be rebuilt, tested, and then deployed. The effort quickly scales out of control.
By finding ways to share components between products, it becomes a lot more feasible to maintain multiple products long term. One way to do this is to bring in standards for how components interact, which ensures interoperability and allows custom products to pull in generic software components where appropriate.
At this point, it is tempting to say, “desktops and servers standards have already solved this problem. Let's use those.” While true, PC platform standards ensure any OS can boot on any hardware in that space. However, it is also simplistic and does not recognize either the hardware constraints or the level of customization required in embedded. We need to handle custom hardware interfaces, and often many components throughout the software stack need to be modified to work in the embedded device. It also does not recognize that the embedded ecosystem is heavily invested in mature technologies like U-Boot and Device tree, and there is a high resistance to migrate away from them. To be useful in embedded, any standards introduced must account for the needs of the embedded ecosystem.
It is also important to understand how the level of standardization impacts the way products are developed. If there is little standardization, like embedded Linux is now, it provides high flexibility on how embedded systems are built, but creates a high maintenance cost over time to support custom developments. On the other hand, strict standards would ensure a high degree of interoperability, but may not provide the flexibility needed by the embedded sector. If standards are too strict, then they are ignored, and they will not do anything to enable new and innovative products. There is a balance between standardization and flexibility, particularly with embedded, which enables innovation and creates better products.
Hitting the right level of standardization for embedded is exactly what the SystemReady IR certification program addresses. SystemReady IR extends the promise of “Just Works” to embedded Linux products by choosing standards that can be implemented using the technology already in use today. A SystemReady IR certification guarantees the platform implements a standard firmware interface that OSes can depend on without customization, and that support for the platform has been merged into mainline Linux.
Or in concrete terms, a SystemReady IR compliant platform can boot generic Linux distributions using the exact same image format and boot flow used by Arm servers and laptops. Using the same boot flow opens all kinds of possibilities for the embedded ecosystem. To list a few, it means that instead of maintaining everything themselves, device vendors can build products using well-supported distributions who have the resources to manage updates and security fixes. It means that custom Yocto images can be booted and tested using virtual machines in the cloud before being deployed out to real hardware. It means engineering effort does not need to be devoted to managing per-platform differences, and, crucially, it becomes easy to support many hardware devices even when building a completely custom OS stack. SystemReady IR means that you can focus on the features that make your product great, and not on individual device enablement problems.
SystemReady IR is the embedded and IoT band of the larger SystemReady program. SystemReady defines and certifies compliance with hardware and firmware standards for Arm systems, with bands for different parts of the Arm ecosystem. The detailed requirements of each band are defined in the SystemReady Requirements Specification (SRS) which is linked to at the end of this article. The list of SystemReady IR certified platforms can be found on the SystemReady web page. The first IR certification was issued in June 2021, and there will be at least 11 certifications published before Arm DevSummit in October 2021, with many more in the certification pipeline.
For standards to be relevant in embedded they need to be two things: They must foster interoperability, and they must work with existing software stacks. Most embedded Linux products are built using U-Boot firmware, a Device tree system description and a custom Linux kernel. U-Boot and Device tree are mature technologies that are widely deployed. However, the way they are used makes interoperability difficult. Deployment is different on every platform due to slight differences in the U-Boot boot scripts. The OS is tied to the platform because the Device tree is tightly coupled to the OS. Keeping the software up to date requires custom code because there is not a standard firmware update interface. So, while the components are capable and mature, lacking standards there is no way to ensure interoperability.
When designing SystemReady IR, we identified the major issues that can be resolved with reasonable design changes. We looked for where we could create a defined interface between the platform (hardware & firmware) and the operating system that would simplify the engineering required. To do that we focused on the specific problems faced by embedded developers. The following table is a helpful summary:
These changes; enabling UEFI, providing DT by default, supporting UpdateCapsule(), and support in mainline solves most of the OS interoperability problems. All the major OS vendors expect the UEFI boot flow and U-Boot already implements UEFI. Enabling it immediately makes U-Boot compatible. Configuring U-Boot to provide a Device tree to the OS by default means that the platform can describe itself to the OS. So, the OS does not need to carry around details about every hardware platform. For generic distributions this is a critical issue because they cannot support per-platform customizations at scale. The OS is generic and cannot contain machine-specific details. Similarly enabling UpdateCapsule() provides a standard interface for updating system components so that the OS does not need to understand a machine-specific update mechanism. Finally, vendor kernels are the biggest pain points in embedded Linux development because it means each product requires a different kernel tree and it makes the platform unsupportable by the Linux distributions. By requiring the hardware be supported in mainline Linux, the distros can ship support for the hardware in their generic images. In addition custom Linux builds can track Linux stable releases instead of depending on a vendor to keep their fork up to date.
Everything listed here is supported today by U-Boot, Trusted Firmware, Linux, the major distros, and Yocto. The configuration options are easy to turn on, and once enabled they open up a wide range of OS and hardware options when building a product. Best of all it does not require replacing any of the software you are already using.
Where SystemReady IR comes in is to formalize the standards discussed previously into a specific set of requirements that silicon vendors and ODMs can implement. It sets up a certification testing program to verify compliance with the specifications. At a high level, the IR band requires that:
Tools, test suites and resources are provided by the IR program to help step through the process of certification. The most important resource is the SystemReady IR: IoT integration, test, and certification guide which covers all the details of how to enable and test for SystemReady IR compliance. The tools are open-source and useful links to them are in the SystemReady IR guide.
The certification process itself is straight-forward. The first step is to contact us and let us know about the platform you want to certify. We ask a few questions to confirm that you can run and pass the certification test procedure, and then put you in contact with one of the SystemReady IR test labs. You need to provide the test lab with a copy of your own test results as well as a sample of the hardware so that the lab can reproduce and confirm your results. Finally, the test lab provides the full results of their testing to Arm. If the platform meets all the requirements and passes testing, then Arm will provide a formal SystemReady certificate and list the platform on the SystemReady IR web page.
The SystemReady IR program is actively testing and certifying boards now. If you have any questions or if you have a product to be certified, then you can contact us at systemready@arm.com. You can read the SystemReady IR Integration, Test, and Certification guide and try running the IR Architecture Compliance Suite (ACS) on your platform. All the program details can be found on our website. Finally, you can help us define the next iteration of SystemReady IR by getting involved with the EBBR community project. Contact and meeting details can be found on the EBBR project page.
Great work Grant!!!