The decreasing cost and power consumption of intelligent, interconnected, and interactive devices at the edge of the Internet are creating massive opportunities to instrument our cities, factories, farms, and environment to improve efficiency, safety, and productivity. Developing, debugging, deploying, and securing software for the estimated trillion connected devices presents substantial challenges. As part of the SMARTER (Secure Municipal, Agricultural, Rural, and Telco Edge Research) project, Arm has been exploring the use of cloud-native technology and methodologies in edge environments to evaluate their effectiveness at addressing these problems at scale.
Smarter-device-manager enables containers deployed using Kubernetes to access devices (device drivers) available on the node. This capability is essential for IoT applications running at the edge because the main function of IoT edge devices is to measure and interact with the external environment. This behavior is not required or desired in a typical cloud installation of Kubernetes. The direct access to device drivers limits the pool of nodes that are able to run those containers and can weaken the security of the system.
IoT applications acquire data about the environment, and possibly control actuators to achieve some desired objective. Systems range in complexity from an internet connected thermostat, to directing traffic using multi-camera video feeds, to very complex industrial process control, such as in chemical plants. The interconnection between the sensors and actuators (IoT endpoints) to IoT Edge gateways can be direct (VIA GPIO, analog I/O, CSI/MSI, and others) or indirect, VIA wireless or wired interfaces (VIA Bluetooth, LoRa, ethernet, USB, and others).
Applications running inside a container do not have access to device drivers unless explicitly given access by Docker or other container runtime engines. In Kubernetes requires external components to allow containers to access devices. Controlled access to these devices is essential to enable a container-based IoT solution, and Smarter-device-manager allows containers to have direct access to devices on the host in a secure and schedulable way. Smarter-device-manager leverages the device plugin API provided by Kubernetes (Kubernetes device plug-ins). This API allows Kubernetes to be informed about the devices that are available on the node, the properties of each device and how to interface with the CRI to enable access to the device when requested. The most important property is related to shareability of the device: how many containers are able to access the same device simultaneously and an enforcement requirements preventing over-subscription of devices.
Smarter-device-manager is a container by itself and is remotely deployable, allowing easy upgrades to different configurations. Multiple copies of the smarter-device-manager can run on a single node allowing precise control of the devices managed and available to containers. The default configuration provided allows devices on a RaspberryPi running raspian/debian/ubuntu to be available to containers.
raspian/debian/ubuntu
The source code of the smarter-device-manager container and examples are available at the repository.
smarter-device-manager repository
The Smarter-device-manager application has four sequential phases of operation.
/dev
The configuration file contains a list of entries, where each entry allows one or more devices to be identified as managed. It also provides metadata that determines how many simultaneous requests can be made for each device that this template matches. Each entry is composed of a regular expression and an integer. The regular expression is used to match the filenames on the "/dev" directory and the integer determines how many simultaneous accesses are allowed. The following entry ("^i2c-*$",1) will match all the I2C interfaces available on "/dev", for example "/dev/i2c-0","/dev/i2c-1". For each device it will create a resource prefixed with smarter/ and with the filename of the device without "/dev/". For example, the resources created for the I2C will be "smarter/i2c-0" and "smarter/i2c-1". The configuration file is matched sequentially and the first entry that matches the filename will be used, whilst the others will be ignored for this filename. The order of reading of the "/dev" directory is not determined. Only the first level of the /dev directory is read, but directories are allowed. For the purpose of the alsa-sound, the directory "/dev/snd" will be matched by the entry ("^snd$",100) and if a container requests the resource "smarter/snd" it will have access to all sound devices on the host. Entries that do not match any files are ignored, allowing multiple nodes with different hardware configurations to be managed by a single configuration file, and only the devices that are available on each node are presented.
"^i2c-*$",1
/dev/i2c-0","/dev/i2c-1
/dev/
smarter/i2c-0
smarter/i2c-1
/dev/snd
"^snd$",100
smarter/snd
The default configuration file is designed to provide all the common Raspberry Pi 2/3/4 devices: sound, GPIO, Bluetooth, I2C, real-time clock, video devices and accelerated video encoder/decoder respectively.
- devicematch: ^snd$ nummaxdevices: 20 - devicematch: ^gpiomem$ nummaxdevices: 40 - devicematch: ^gpiochip[0-9]*$ nummaxdevices: 20 - devicematch: ^hci[0-9]*$ nummaxdevices: 1 - devicematch: ^i2c-[0-9]*$ nummaxdevices: 1 - devicematch: ^rtc0$ nummaxdevices: 20 - devicematch: ^video[0-9]*$ nummaxdevices: 20 - devicematch: ^vchiq$ nummaxdevices: 20 - devicematch: ^vcsm.*$ nummaxdevices: 20
Smarter-device-manager is deployable by Kubernetes itself and is compatible with k3s and k8s. The following YAML file can be used as an example
apiVersion: v1 kind: Pod metadata: name: smarter-device-management namespace: default spec: priorityClassName: "system-node-critical" hostNetwork: true dnsPolicy: ClusterFirstWithHostNet hostname: smarter-device-management nodeName: <replace with node to run> containers: - name: smarter-device-manager image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:20191204204613 imagePullPolicy: IfNotPresent securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] resources: limits: cpu: 100m memory: 10Mi requests: cpu: 10m memory: 10Mi volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins - name: dev-dir mountPath: /dev volumes: - name: device-plugin hostPath: path: /var/lib/rancher/k3s/agent/kubelet/device-plugins - name: dev-dir hostPath: path: /dev terminationGracePeriodSeconds: 30
This example shows how a Raspberry Pi 3 running Ubuntu 19.10 presents the resources to k8s or k3s. The following command:
kubectl describe node pike5
Produces the following output:
Name: pike5 Roles: <none> Labels: beta.Kubernetes.io/arch=arm beta.Kubernetes.io/os=linux smarter-device-manager=enabled Annotations: node.alpha.Kubernetes.io/ttl: 0 CreationTimestamp: Mon, 02 Dec 2019 09:22:56 -0600 Taints: <none> Unschedulable: false Lease: HolderIdentity: <unset> AcquireTime: <unset> RenewTime: <unset> Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Thu, 16 Jan 2020 08:20:06 -0600 Mon, 02 Dec 2019 09:22:56 -0600 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 16 Jan 2020 08:20:06 -0600 Wed, 04 Dec 2019 09:47:08 -0600 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 16 Jan 2020 08:20:06 -0600 Mon, 02 Dec 2019 09:22:56 -0600 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 16 Jan 2020 08:20:06 -0600 Mon, 16 Dec 2019 14:58:05 -0600 KubeletReady kubelet is posting ready status. AppArmor enabled Addresses: InternalIP: XXX.XXX.XXX.XXX Hostname: pike5 Capacity: cpu: 4 ephemeral-storage: 14999512Ki memory: 873348Ki pods: 110 smarter-devices/gpiochip0: 10 smarter-devices/gpiochip1: 10 smarter-devices/gpiochip2: 10 smarter-devices/gpiomem: 10 smarter-devices/i2c-1: 10 smarter-devices/snd: 10 smarter-devices/vchiq: 10 smarter-devices/vcs: 0 smarter-devices/vcsm: 10 smarter-devices/vcsm-cma: 0 smarter-devices/video10: 0 smarter-devices/video11: 0 smarter-devices/video12: 0 smarter-devices/video4: 10 Allocatable: cpu: 4 ephemeral-storage: 13823550237 memory: 770948Ki pods: 110 smarter-devices/gpiochip0: 10 smarter-devices/gpiochip1: 10 smarter-devices/gpiochip2: 10 smarter-devices/gpiomem: 10 smarter-devices/i2c-1: 10 smarter-devices/snd: 10 smarter-devices/vchiq: 10 smarter-devices/vcs: 0 smarter-devices/vcsm: 10 smarter-devices/vcsm-cma: 0 smarter-devices/video10: 0 smarter-devices/video11: 0 smarter-devices/video12: 0 smarter-devices/video4: 10 System Info: Machine ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX System UUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Boot ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Kernel Version: 5.3.0-1014-raspi2 OS Image: Ubuntu 19.10 Operating System: linux Architecture: arm Container Runtime Version: docker://19.3.2 Kubelet Version: v1.13.5 Kube-Proxy Version: v1.13.5 Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- argus smarter-device-manager-gdmjk 10m (0%) 100m (2%) 15Mi (1%) 15Mi (1%) 43d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 560m (14%) 850m (21%) memory 365Mi (48%) 365Mi (48%) ephemeral-storage 0 (0%) 0 (0%) smarter-devices/gpiochip0 0 0 smarter-devices/gpiochip1 0 0 smarter-devices/gpiochip2 0 0 smarter-devices/gpiomem 0 0 smarter-devices/i2c-1 0 0 smarter-devices/snd 0 0 smarter-devices/vchiq 1 1 smarter-devices/vcs 0 0 smarter-devices/vcsm 1 1 smarter-devices/vcsm-cma 0 0 smarter-devices/video10 0 0 smarter-devices/video11 0 0 smarter-devices/video12 0 0 smarter-devices/video4 2 2 Events: <none>
It is recommended to use a set of daemonSet to deploy the smarter-device-manager on nodes, where each daemonSet contains the configuration for a specific hardware configuration if necessary.
The following YAML file describes a daemonSet that creates a smarter-device-manager at each node that has the label "smarter-device-manager" with the value "enabled".
apiVersion: v1 kind: Namespace metadata: name: < Replace with the namespace to use > labels: name: < Replace with the namespace to use > --- apiVersion: apps/v1 kind: DaemonSet metadata: name: smarter-device-manager namespace: < Replace with the namespace to use > labels: name: smarter-device-manager role: agent spec: selector: matchLabels: name: smarter-device-manager updateStrategy: type: RollingUpdate template: metadata: labels: name: smarter-device-manager annotations: node.Kubernetes.io/bootstrap-checkpoint: "true" spec: nodeSelector: smarter-device-manager : enabled priorityClassName: "system-node-critical" hostname: smarter-device-management hostNetwork: true dnsPolicy: ClusterFirstWithHostNet imagePullSecrets: - name: k8sedgeregcred containers: - name: smarter-device-manager image: registry.gitlab.com/arm-research/smarter/smarter-device-manager/smarter-device-manager:20191204204613 imagePullPolicy: IfNotPresent securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] resources: limits: cpu: 100m memory: 15Mi requests: cpu: 10m memory: 15Mi volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins - name: dev-dir mountPath: /dev volumes: - name: device-plugin hostPath: path: /var/lib/rancher/k3s/agent/kubelet/device-plugins - name: dev-dir hostPath: path: /dev terminationGracePeriodSeconds: 30
The configuration file can be changed by using a configmap and replacing the file "/root/config/conf.yaml" on the container.
/root/config/conf.yaml
Smarter-device-manager does not rename devices and the same filename existing on the host will be present on the "/dev" container. This has implications on how the container can find out which device to use if more than option is available. For example, if multiple I2C interfaces are available on the host, but only one is enabled on a container, that container may have the /dev/i2c-1 instead of dev/i2c-0. The application container can discover what devices are available by an external configuration of the container, environment variable, command line changes or a configuration file changes but all of those require changes on the YAML configuration of the container. If a single device is needed for the container, a simple self-discovery mechanism is available since only one file will be present on the "/dev/", allowing the application to pick whatever device is available on the running container. In the previous example of I2C, the container just uses whatever file is present; "/dev/i2c-*" that will find the "/dev/i2c-1" as shown before.
/dev/i2c-1
dev/i2c-0
/dev/i2c-*
Some devices need to be allocated together to provide the correct or full functionality. For example, the video encoder/decoder on a Raspberry Pi requires two devices (/dev/vcsm and /dev/vcsm-cma). Smarter-device-manager does not support enforcement of that rule, so all the devices have to be requested by the container in order for the correct operation to be achieved.
/dev/vcsm
/dev/vcsm-cma
The plugin API from Kubernetes prevents the removal of a resource even if the provider of the resources (smarter-device-manager) is not running. However, it makes the resource unavailable by setting the number of available resources to zero. The availability of resources is verified only when the container starts, so if the resources are no longer available the container is not stopped. The following example shows that the video4 device is still available (10 units) but video10,11 and 12 are no longer available, even though they are still listed.
Capacity: cpu: 4 ephemeral-storage: 14999512Ki memory: 873348Ki pods: 110 smarter-devices/gpiochip0: 10 smarter-devices/gpiochip1: 10 smarter-devices/gpiochip2: 10 smarter-devices/gpiomem: 10 smarter-devices/i2c-1: 10 smarter-devices/snd: 10 smarter-devices/vchiq: 10 smarter-devices/vcs: 0 smarter-devices/vcsm: 10 smarter-devices/vcsm-cma: 0 smarter-devices/video10: 0 smarter-devices/video11: 0 smarter-devices/video12: 0 smarter-devices/video4: 10
Smarter-device-manager enables IoT applications managed by Kubernetes to precisely control of which devices each container can access providing better security and resource management. The current implementation has some limitations that originates from the host OS, container management runtime or Kubernetes. Here are some examples of current limitations and workarounds.
In the current implementation, the smarter-device-manager scans the /dev only when starting up, so devices that are connected after the container is started will not be seen. The workaround is restarting the device manager container so the new devices will be added to the available resources.
Smarter-device-manager exports the controller but not the devices. For some interfaces (i.e. SPI, I2C, Bluetooth) access to the controller is managed as a resource, but any container that has access to the controller will access any device that can interface with that controller. In the current implementation the applications are responsible to manager access to devices, the smarter-device-manager only provide access control to the bus controller.
An application container sees the same devices as the host, for example the “/dev/video1” on the host will be “/dev/video1” on the container even though on the container the “/dev/video0” may not exists, so an application has to scan the directory to determine which camera to use instead of always using the same device. In the current implementation, it is not possible to request any camera or video source, but only a specific one. In some cases, some form of hardware discovery that allows the application containers to be dynamically configured can be useful.
/dev/video1
/dev/video0
Contact Alexandre Ferreira
This post is the third in a five part series. Read the other parts of the series using the following links: Part one: SMARTER: A smarter-cni for Kubernetes on the Edge Part two: SMARTER: An Approach to Edge Compute Observability and Performance Monitoring Part four: SMARTER: Debugging a Remote Edge Device
This post is the third in a five part series. Read the other parts of the series using the following links:
Part one: SMARTER: A smarter-cni for Kubernetes on the Edge
Part two: SMARTER: An Approach to Edge Compute Observability and Performance Monitoring
Part four: SMARTER: Debugging a Remote Edge Device
Thank you. Your suggestion on how to use the devices is correct. I added an example that you can use to test, it uses alsa but can be modified to use ttyACM0. Your error indicates that even though the smarter-devices/ttyACM0 exists there is no allocatable units in the nodes where the container can run. Can you check the node if there are allocatable smarter-devices/ttyACM0 units for that device?
apiVersion: v1 kind: Pod metadata: name: smarter-device-management-client namespace: NAMESPACE spec: serviceAccountName: default automountServiceAccountToken: false dnsPolicy: ClusterFirstWithHostNet hostname: test-client nodeName: NODE_TO_TEST restartPolicy: Never containers: - name: smarter-device-management-client imagePullPolicy: IfNotPresent image: alpine command: ["/bin/ash"] args: - "-c" - | if [ ! -d /dev/snd ] then echo "No sound directory available (/dev/snd)" exit 1 fi apk add alsa-utils if [ $? -gt 0 ] then echo "Could not install alsa-utils" for i in 1 2 3 4 5 6 7 8 9 10 do sleep 20 done exit $? fi if [ $? -gt 0 ] then echo "Could not install alsa-utils" exit $? fi RESULT=$(aplay -L) if [ $? -gt 0 ] then echo "Could not execute aplay" exit $? fi NL=$(echo "${RESULT}" | grep tegrasndt19xmob | wc -l) if [ ${NL} -ne 2 ] then echo "Aplay did not find the correct device check:" echo "${RESULT}" exit 11 fi exit 0 resources: limits: cpu: 100m memory: 100Mi smarter-devices/snd: 1 requests: cpu: 100m memory: 100Mi smarter-devices/snd: 1 terminationGracePeriodSeconds: 10
Hi, great tutorial. thank you.
I was wondering how do you consume a device in a container?
I have tried adding a request/limit under resources, this leads to "0/1 nodes are available: 1 Insufficient smarter-devices/ttyACM0"
when I describe the node I can see smarter-devices/ttyACM0 listed as a resource