SMARTER: Debugging a Remote Edge Device

The decreasing cost and power consumption of intelligent, interconnected, and interactive devices at the edge of the Internet are creating massive opportunities to instrument our cities, factories, farms, and environment to improve efficiency, safety, and productivity. Developing, debugging, deploying, and securing software for the estimated trillion connected devices presents substantial challenges. As part of the SMARTER (Secure Municipal, Agricultural, Rural, and Telco Edge Research) project, Arm has been exploring the use of cloud-native technology and methodologies in edge environments to evaluate their effectiveness at addressing these problems at scale.

Edge devices are typically behind firewalls, so it is not possible to connect to them with SSH. Also, there may be multi-tenancy: it might not be possible to grant root access on the device to the developer of an application running on the device. Before deploying an application to a remote edge device, a developer will typically test it on a machine to which they have physical, or at least, shell access. However, a problem might appear only after the application has been deployed remotely. What should the developer then do if the problem cannot be reproduced locally and a direct SSH connection to the remote edge device is not possible?

One solution is presented here. A special application, a privileged “debug container”, is deployed to the edge device, and this container connects to a “comms” service deployed on a publicly accessible system such as a cloud server. The developer also connects to the comms service and has their connection forwarded to the edge device. The net result is to provide the developer with something that closely resembles ssh to the edge device or docker exec to containers running on the device, but which works despite firewalls not allowing a direct connection. Moreover, by relying on host binaries mounted into it, the “debug container” image is tiny and architecture independent.

Prototype code for doing this has been published on GitLab:

Prototype code on GitLab 

In the rest of this article I will first give an overview of the design, and then show how to deploy the prototype code to demonstrate it working.

How it works

The implementation is based on SSH, as that provides both secure connections and terminal handling, both of which one would prefer not to reimplement. It seems to be easier and more secure to use three SSH connections:

• From the edge device to the comms service
• From the developer's machine to the comms service,
• One from the developer's machine through a tunnel, created by the other two connections, all the way to the edge device.

The comms service may be shared by multiple developers and multiple connections, so it has a REST API for setting up accounts and uploading SSH keys. As a consequence, it uses two ports for incoming connections: one for HTTP and one for SSH.

When the comms service has been set up and configured, the debug container can be deployed to the edge device. This connects to the comms service using SSH with port forwarding from a Unix-domain socket inside the comms container to an SSH daemon running inside the debug container.

The developer can now connect using a Perl script that invokes the SSH client twice, like this: ssh -o "ProxyCommand ssh ..." ...

The inner ssh invocation, inside the ProxyCommand, connects to the comms service, where a login shell invokes socat to forward the connection to the Unix-domain socket created by the connection from the debug container. The outer ssh invocation then connects through the pipe thus created through to the SSH daemon running in the debug container, where another special login script is executed. This will execute either nsenter to run a shell, command on the host, docker to run a shell, or command in another container.

This is a simplified description. To learn more about the details, please refer to the prototype code. For example, there are facilities to make an account on the comms service, but these will expire at a particular time in the future, and the developer can run a particular command remotely by specifying it on the command line. This is then base64-encoded so that the overall behaviour is more like docker exec than like ssh, which handles its command-line arguments by concatenating them with spaces and feeding them to /bin/sh, with the consequence that double shell-quotation is sometimes required.

The security of the edge device depends primarily on two things. Firstly, how the debug container is configured: whether it grants root access to the system or docker exec access to particular containers, and what public keys for authentication are included (~/.ssh/authorized_keys). And, secondly, who has the corresponding private keys for authentication to the debug container. The other two key pairs used in the system are for authentication to the comms container. An attacker who acquires those could abuse the comms container, but should not be able to gain access to the edge device.

Deploying the prototype code

For a compelling demo, you would use three machines – the “edge device”, the “comms server”, and the “developer's machine”. The “edge device” and the “developer's machine” would be behind firewalls. However, the following instructions should work just the same if you are using one or two machines to act as the previous three, and the firewalls are, of course, not required in order for it to work.

Clone the remote-debug repo onto each machine that you are using:

git clone https://gitlab.com/arm-research/smarter/remote-debug
cd remote-debug

The first step is to build and deploy the comms container. For practical use the comms container should be deployed on some cloud infrastructure with a front end to perform authentication. However, if you just want to test, or you want to debug the comms service, you can run it on any machine to which you have shell access like this:

docker build -t comms comms
docker run -it -p 10022:22/tcp -p 10080:80/tcp comms

This makes the service available to anyone who can access ports 10022 and 10080 on that machine, so you would probably use a machine on your local network.

To deploy the comms container on cloud infrastructure, you would probably build the Docker image and push it to a registry roughly like this:

docker login registry.gitlab.com
docker build -t registry.gitlab.com/.../debug-comms:20200430 comms
docker push registry.gitlab.com/.../debug-comms:20200430
docker logout registry.gitlab.com

Then run the comms container on the cloud infrastructure with kubectl:

cat > comms.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: debug-comms
spec:
  selector:
    matchLabels:
      app: debug-comms
  replicas: 1
  template:
    metadata:
      labels:
        app: debug-comms
    spec:
      nodeSelector:
        debug-node: "yes"
      containers:
      - name: debug-comms
        image: registry.gitlab.com/.../debug-comms:20200430
        ports:
        - containerPort: 22
      imagePullSecrets:
      - name: ...
---
kind: Service
apiVersion: v1
metadata:
  name: debug-comms
spec:
  selector:
    app: debug-comms
  ports:
  - name: debug-port
    protocol: TCP
    port: 222
    targetPort: 22
  clusterIP: ...
  externalIPs: [...]
EOF
kubectl apply -f comms.yaml

The standard SSH port 22 has been mapped to 222 here, as 22 is often reserved, but HTTP access is left at 80. You must now make the cloud server's ports 80 and 222 visible to the outside world, with 80 going through some authentication layer. This typically involves interacting with a vendor-specific web interface.

When you have deployed the comms container, the next step is to generate three key pairs. This is best done on the developer's machine. If that system has a sufficiently recent SSH (for example, OpenSSH 6.5 of Jan 2014) then we can use Ed25519:

ssh-keygen -t ed25519 -C client -P '' -m PEM -f client
ssh-keygen -t ed25519 -C server -P '' -m PEM -f server
ssh-keygen -t ed25519 -C dev -P '' -m PEM -f dev

The server key is used for the edge device to connect to the comms container, the client key is used for the developer's machine to connect to the comms container, and the dev key is used to connect to the edge device through the tunnel created with the other two connections. So the server key is provisioned into the debug container (see the following example) and the other two keys are provided (securely) to the developer. It is the dev key which is critical for the security of the edge device.

The comms container needs the server and client public keys. We provide them using the comms container's REST API. Run this on the developer's machine or wherever the keys were generated:

curl -L -X POST -H 'Content-Type: application/json' \
  -H 'X-Auth-Account-Id: boss' -d \
  '{ "user":"bob", "expiry": "'$(date -u -d '2038-01-20' '+%s')'",
     "ckey": "'"$(cat client.pub)"'", "skey": "'"$(cat server.pub)"'"}' \
  http://localhost:10080/create_user

The response should be {"user":"bob"}, indicating that we were given the username we asked for. If we had omitted the "user":"bob" from the JSON then a username would have been invented for us.

We can now build and deploy the debug container onto our edge device.

The debug container image is rather unusual. It contains no binaries and relies on host binaries being mounted into it. It is therefore architecture-independent, but not OS-independent; the host system must have certain versions of certain programs in the expected places, for example /usr/bin/sshd. An advantage of this approach is that the image is tiny and can be deployed onto a resource-limited device without perturbing it much.

Build the debug image and upload it to a registry like this:

docker login registry.gitlab.com
docker build -t registry.gitlab.com/.../debug:20200430 debug
docker push registry.gitlab.com/.../debug:20200430
docker logout registry.gitlab.com

If you wanted to bypass the registry, you could build the image on the edge device:

docker build -t debug:20200430 debug

Either way, you can now deploy the debug container using Kubernetes. If you are using a registry you may need to first configure the edge device with whatever secrets are required to pull from the registry.

Since some of the potential substitutions are non-trivial, there is a script for generating the YAML for Kubernetes. The debug container requires two of the SSH keys generated previously, and the values 10022 and bob come from how the comms container was configured. NODENAME must be replaced with the name of the edge device, and COMMSHOST with the address of the machine on which the comms container was deployed. The debug-yaml script requires the keys, so run that command on the developer's machine or wherever the keys were generated. If because of the way your Kubernetes is set up you need to run the kubectl on a different machine, just copy the debug.yaml to that machine first.

In this example we will be granting shell access to the remote device so there is no containerid variable set. The generated debug.yaml will specify privileged access for the debug container.

debi=debug:20200430 node=NODENAME host=COMMSHOST port=10022 user=bob \
  skey="$(cat server)" dkey="$(cat dev.pub)" ./debug-yaml > debug.yaml
kubectl apply -f debug.yaml

If the debug container was successfully deployed, it should now be possible to connect from the developer's machine to the edge device using the rdebug script provided. Replace COMMSHOST as previous and run this on the developer's machine:

./rdebug bob COMMSHOST 10022 client dev

If everything has gone to plan, that should give you a shell prompt on the edge device.

If you want to test configuring access to a particular container on the edge device, first obtain the target container's ID using kubectl and then add containerid=... to the environment of debug-yaml when generating the YAML for deploying the debug container. It is possible to deploy several differently configured debug containers onto the same device, for example to grant access by two different developers to two different containers.

Conclusion

Our approach exploits facilities that are already in use, namely Kubernetes, Docker, and a standard Linux distribution. It is hoped that this solution, built from existing tools, will facilitate the deployment of containerised applications to edge devices, which is a key goal of the SMARTER project. Another approach appropriate for larger scale deployments is offered through Arm Pelion’s Secure Device Access (SDA), which also gives fine-grained control over permissions and can be provisioned through the Pelion device management portal. 

Contact Edmund Grimley-Evans 

This post is the fourth in a five part series. Read the other parts of the series using the following links:

Part one: SMARTER: A smarter-cni for Kubernetes on the Edge

Part two: SMARTER: An Approach to Edge Compute Observability and Performance Monitoring

Part three: SMARTER: Smarter-Device-Manager for Kubernetes on the Edge