The decreasing cost and power consumption of intelligent, interconnected, and interactive devices at the edge of the Internet are creating massive opportunities to instrument our cities, factories, farms, and environment to improve efficiency, safety, and productivity. Developing, debugging, deploying, and securing software for the estimated trillion connected devices presents substantial challenges. As part of the SMARTER (Secure Municipal, Agricultural, Rural, and Telco Edge Research) project, Arm has been exploring the use of cloud-native technology and methodologies in edge environments to evaluate their effectiveness at addressing these problems at scale.
As cloud computing has become more pervasive, services providing new run-time options and programming models have emerged. One of these newer programming models is Function as a Service (FaaS). The evolution of server virtualization technologies started with the first virtual machines (VMs), which allowed server equipment to be utilized more efficiently. VMs, once the de-facto method of deploying applications on servers, require the user to install the OS and to manage every service running on it, leading to maximum control, but significant management costs. Containers became the next step in server resource virtualization, where developers encapsulate their application logic inside containers and the cloud provider is in charge of providing the OS and the networking. While containers significantly reduce the management overheads, they still require the application owner to deal with scaling and controlling when and where the containers run, often through a proxy, like Kubernetes. Serverless computing is the latest evolution of server resource virtualization. The idea behind serverless is that the developers should only focus on their application, and they should let the cloud provider worry about infrastructure, like setting up operating systems, network configuration, scheduling, and scaling-out. Using FaaS to build applications leads to a Serverless architecture. The FaaS model offers several attractive features for application developers: on-demand execution, automatic scaling, pay per invocation, and transparent infrastructure maintenance. While some applications run best on bare-metal, VMs or containers, a significant number of players have moved many (or all) components of their software stack to a FaaS model due to their simplicity and pay-per-invocation scheme. Spearheaded by AWS Lambda, the FaaS model quickly grew in popularity, and, today, every major cloud provider offers a FaaS service.
One of the goals of the SMARTER project is to provide an edge programming environment that is similar to the cloud environment developers have become accustomed to. The EdgeFaaS project aims to provide a FaaS environment optimized for the wide-range and unique configurations where SMARTER is deployed, while providing the familiar FaaS feel and advantages.
Instead of writing our FaaS runtime from scratch, we studied several open-source projects that we could use as a base, specifically OpenWhisk, Fission, Kubeless, and OpenFaaS. Despite Fission and OpenFaaS being really close to what we need, their architecture assumes a traditional Kubernetes deployment where the whole cluster is fully connected with stable, high-bandwidth links.
One of our aims for EdgeFaaS is to minimize the footprint of adding new functions to edge devices. Edge devices tend to be limited in memory and storage, and some are located in remote locations, connected to the rest of the world VIA slow, expensive cell network connections. This desire for lightweight function installation led to the modification of the OpenFaaS runtime to support multiple functions per container. With this modification, installing a new function is as small as the text file containing the function code, whereas EdgeFaaS's original optimized Python function containers were at least 50MB in size.
We based our EdgeFaaS solution on OpenFaaS. OpenFaaS is an active open source FaaS solution. It is relatively simple, and it supports multiple target architectures and multiple programming languages. OpenFaaS was designed as a lightweight FaaS environment targeting Kubernetes clusters of IoT devices. This was a huge plus for us because the project already supported both the armhf and the arm64 architectures most edge devices run on. We reused the watchdog component and modified it to be able to share multiple functions and concurrent invocations. We also support on-the-fly function installation and deletion with minimal network traffic. We did not use most of the OpenFaaS components/features (like load-balancing and auto-scaling) because these would not work with the constrained SMARTER network topology.
armhf
arm64
watchdog
EdgeFaaS System Architecture
We created a new component, EdgeFaaS Manager, to control the installation, listing, and deletion of EdgeFaaS functions. Users can directly manage EdgeFaas (as described in the following), but this is not scalable nor possible in more realistic deployments where thousands nodes live behind firewalls.
Two of our differentiating characteristics from OpenFaaS are that we run one runtime per tenant per language and we can deploy multiple functions per container without rebuilding or restarting the container.
Three of our differentiating characteristics from OpenFaaS are:
To reduce the attack surface, the EdgeFaaS runtime runs in normal user mode without root priveleges, and the function files reside in the app user's home directory. The following is the directory structure of an EdgeFaaS container with two functions, echo & other_handler.
echo
other_handler
/home/app # tree . . └── function ├── echo │ ├── __init__.py │ ├── handler.py │ ├── handler.pyc │ └── index.py └── other_handler ├── __init__.py ├── handler.py ├── handler.pyc └── index.py 3 directories, 8 files
Each directory under function is named after the function it executes and each of them contains the same files.
index.py
handler.py
fwatchdog
handle
The fwatchdog binary is the main EdgeFaaS program. It runs an HTTP server that handles function invocations and function management requests. For function invocations, the fwatchdog checks if the request is valid and forks a new process. This new process calls index.py , which calls the actual function. For management requests, the fwatchdog executes the task. The following graph outlines the server's program flow.
The target platform of our project are edge and IoT devices, so we chose the Raspberry Pi 4, a widely available, small development board to develop the EdgeFaaS runtime. The Raspberry Pi 4 dev board we used features a Quad core Cortex-A72 SoC and 2GB of RAM. Our development board was running Raspbian 10 Buster with Linux 4.19. We used Docker 19.03.4 with experimental features enabled.
One of our overarching goals for this and related projects was to automate as much of the build and testing processes as possible. Throughout the development of EdgeFaaS, I scripted the build process as much as possible. Since my goal is to make this article as accessible and educational as possible, I have included more detailed steps to explain what's happening. Note: if you're planning to run EdgeFaaS using Kubernetes, you would need to push your image to a registry, as explained in sections Building and hosting the image remotely using GitLab and Building the image locally and pushing it to a container registry.
If your goal is just to build the image as quickly as possible on your local development machine, there is a script that automates the process. To use the build.sh script for a local build, make sure you do not pass the -r argument. The default version of the script builds for armhf, but you can build for other architectures by setting the -a argument to arm, arm64 or amd64 (./build.sh -a amd64).
build.sh
-r
arm, arm64
amd64 (./build.sh -a amd64).
$ ./build.sh <truncated output> $ docker images | grep latest watchdog latest cda305414bec 15 minutes ago 65.1MB
You can inspect the script if you want to modify any of the intermediate steps. The script has unnecessary steps for a simple local build, but I made it this way so it fits several use-cases.
To run EdgeFaaS, skip to Launching the EdgeFaaS runtime.
As explained earlier, I divided the build process into multiple Docker images to decrease the image size and to increase independence (and parallelism) between different parts of the build process.
Using Dockerfile.func, create an image that includes: the minimal Python runtime, the correct environment variables, the default echo function, a non-root app user (to increase security), the expected directory structure, the necessary pip packages.
Dockerfile.func
$ docker build --platform linux/$TARGETARCH -t edgefaas:part . -f Dockerfile.func <truncated output>
Note: I specified linux/$TARGETARCH as the platform so Docker chooses the correct image from the available ones in the manifest. The --platform flag is an experimental feature that allows a multi-platform host to build images for other platforms. If we're not cross-building, this is not necessary, but it's good practice to be explicit.
linux/$TARGETARCH
--platform flag
This step creates a Docker image that includes everything needed, except the watchdog code that is at the center of this implementation of EdgeFaaS.
To create the watchdog binaries, we need an environment with all the GoLang tools and settings needed to compile the source code. We generate the binary inside a builder container. Compiling the code inside a builder container has the following benefits: increased reproducibility due to a tightly controlled build environment, high build independence from the host machine, and minimal modifications to the host machine.
Create the watchdog binaries by building the builder container, specified in Dockerfile.wdog
Dockerfile.wdog
$ docker build -t edgefaas:wdog . -f Dockerfile.wdog <truncated output>
Note: the --platform flag is left unspecified because the host-native Go compiler is faster and can do cross-compilation.
The resulting Docker image is pretty large (at least 500MB) because it contains the Go tools. We could have made the tooling/building/cleanup one layer (to keep the size down), but having a layer with the tools cached makes development quicker.
The beauty of the builder container is that we just extract the files we care about (in this case the fwatchdog-$TARGETARCH binary) and the builder container itself never touches the target edge device. To be able to extract files from the container, first create the container.
fwatchdog-$TARGETARCH
$ docker create --name builder edgefaas:wdog echo $ docker cp builder:/go/src/github.com/openfaas/faas/watchdog/watchdog-$TARGETARCH ./fwatchdog-$TARGETARCH
Now the platform-specific watchdog binary resides in our host work directory. Feel free to remove the builder container since we took what we needed.
$ docker rm builder
The last step to create the EdgeFaaS Docker image is to combine the Python runtime image with the watchdog binary. For this we use the runtime image as base, then we copy in the watchdog and set the default command CMD to run the watchdog. Use Dockerfile.full to achieve this.
CMD
Dockerfile.full
$ docker build --platform linux/$TARGETARCH --build-arg IMAGE_PART=edgefaas:part --build-arg TARGETARCH=$TARGETARCH edgefaas . -f Dockerfile.full <truncated output>
Note: we used --platform to make sure the base image is the desired one (there could be multiple platforms for the images we built). We pass the base image name as an argument using --build-arg because the full name of the image uses the registry URL when building non-locally, so I did not want to hardwire that inside the Dockerfile.
--platform
--build-arg
At this stage, you should have the edgefaas:latest image built.
edgefaas:latest
$ docker images | grep edgefaas REPOSITORY TAG IMAGE ID CREATED SIZE edgefaas latest 191a98c01a4d 37 seconds ago 65.1MB edgefaas wdog 88efca915d28 50 minutes ago 590MB edgefaas part c5948fe95afb 2 hours ago 53.4MB
If you are eager to run your newly-built EdgeFaaS, feel free to skip to Launching the EdgeFaaS runtime.
One of the big appeals of developing applications that run inside containers is that containers are highly portable. Software developers can build containers in their development machine, and the same container can be run in a testing environment or deployed to the target device. The task of copying containers can be done manually (e.g. using scp), but this becomes impossible to scale when deploying containers to a large number of machines. Container registries solve this problem. Container image registries, like Docker Hub, store container images in a place where the machines needing them can access them.
scp
The build.sh script was designed to push the EdgeFaaS images to the desired container registry. I designed it so it pushes the partial and builder containers into the registry. This is useful because these registry images (and their layers) can be used as caches, speeding up the build process.
If you would like to push your images to a registry, make sure to log in.
$ docker login registry.example.com:<optional_port> <truncated output>
The build.sh script accepts the following optional arguments:
-r: OPTIONAL.
./build.sh -r registry.example.com:4567/group/edgefaas
-a: OPTIONAL.
arm (armhf)
arm, arm64 and amd64.
./build.sh -a amd64
We can now run the script.
$ ./build.sh -r registry.example.com:4567/group/edgefaas -a arm <truncated output> $ docker images | grep latest registry.example.com:4567/group/edgefaas latest dd05346b10c4 34 minutes ago 65.1MB
Note: if the images did not exist in the registry, you might see an error that the manifest for the image was not found. This error is safe to ignore and the script should continue running.
The build.sh script is designed to tag images with the git repo's commit hash. The final image tag is just this hash, but the tags of the partial and builder containers end in part and wdog respectively. The script also tags the most recently build images with the following tags: <BRANCH> for the full container, <BRANCH>-part for the partial container, and <BRANCH>-wdog for the builder container.
wdog
<BRANCH>
<BRANCH>-part
<BRANCH>-wdog
To further automate the build process for this project, I integrated it with GitLab's Continuous Integration (CI) service. Our GitLab is setup with GitLab Runners running in Docker containers on amd64 machines. These jobs run in Docker-in-Docker with experimental features enabled. To learn more about our setup see Eric's blog post.
This code repository is also hosted on GitLab and it is setup such that the CI jobs, specified in the repo's .gitlab-ci.yml file, run when a new commit is pushed into GitLab.
.gitlab-ci.yml
For the CI file, I used the same build structure used in the previous sections, but I set it up in a way that the partial and builder containers can run in parallel. I also used the caching techniques (pushing intermediate images and pulling from the registry before building) used in the convenience script. Since the GitLab Runner jobs are launched from the same GitLab instance hosting the source code repo, the login is done within the script using a password automatically filled by the service.
Since a new container is created for each step, no stage is shared by each build. This is problematic because we could like to pass the watchdog binary from the builder stage (watchdog) to the final stage (full-func). This is solved by specifying that the binary is an artifact generated in the builder stage, and by specifying that the final stage has a dependency on the builder stage. Please search for artifacts and dependencies in the .gitlab-ci.yml file.
full-func
To reduce complexity, our EdgeFaaS system runs as a standalone Docker container. Running as a single container reduces scheduling and networking difficulties when deploying EdgeFaaS. Multiple EdgeFaaS containers can independently run in the same host machine.
Since EdgeFaaS runs as a single container, it can be directly deployed using Docker. For bigger deployments, EdgeFaaS containers can be deployed using Kubernetes. We provide sample .yaml files that work for Kubernetes deployments.
.yaml
The EdgeFaaS runtime is built into a Docker container, so users can directly use the Docker runtime to run EdgeFaaS. We setup EdgeFaaS so it is easy to set up by hand. Running EdgeFaaS via Docker is great for developing code and for small-scale deployments where you have access to the target machines.
In this section, I will guide you through the steps of launching EdgeFaaS into a single node. We will focus on the simplest case; if you need to run a more customized setup, please feel free to build upon this guide. Throughout this guide, I will assume the images reside in a container registry so I will use the URL for the containers. If you built EdgeFaaS on the target host, you can use the image name (usually edgefaas) instead of the URL to deploy the container.
edgefaas
There are no Docker images required other than the ones we explicitly download.
$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES $ docker images -a REPOSITORY TAG IMAGE ID CREATED SIZE
Launch the container.
$ docker run --name edgefaas -d --rm registry.gitlab.com/arm-research/smarter/edgefaas Unable to find image 'registry.gitlab.com/arm-research/smarter/edgefaas:latest' locally latest: Pulling from arm-research/smarter/edgefaas <truncated output> Digest: sha256:5b78ca0b5762d46dff529ecc2ca7c0bae37071291818ee3b2c88ac46bd890e35 Status: Downloaded newer image for registry.gitlab.com/arm-research/smarter/edgefaas:latest 662e00dac7ec975e876d8f20bdbc961a3198fdc081e649dff2c107b87d059d85
Note: I named the container edgefaas so it is easy to refer to it. We ran in detached (-d) mode because I am not interested in the log messages. These can be useful if debugging the runtime or new functions. I passed --rm because I would like the container to be cleaned up when it exits. These flags are optional.
-d
--rm
Now, let us find the container's IP address. I'll store it as an environment variable to not have to retype it. Since we did not expose any ports, this container is only accessible from the host machine, but you can set it up differently.
$ docker inspect edgefaas | grep -i ipaddress "SecondaryIPAddresses": null, "IPAddress": "172.17.0.2", "IPAddress": "172.17.0.2", $ FAAS=172.17.0.2
Test the EdgeFaaS runtime using the default echo function. The default port is 8080.
8080
$ curl $FAAS:8080/echo -d "Welcome to EdgeFaaS!" Welcome to EdgeFaaS!
To manage functions, feel free to skip to Managing Functions.
EdgeFaaS was developed with Kubernetes in mind. At Arm Research, we have been exploring what an edge deployment of cloud services looks like, so supporting Kubernetes, the pervasive container orchestration system started by Google and supported by every cloud, is an obvious goal. We have been experimenting with multiple edge-oriented Kubernetes distributions, like Rancher Labs' k3s.
k3s
The following steps should work on any Kubernetes-compliant distribution (please let us know if they do not). But, they were tested on k3s. I chose k3s because it was developed with Arm Edge devices in mind, so it is stripped down to the bare minimum. I also recommend kubeadm for a simple-to-deploy, official Kubernetes distribution. Here are the commands I used to start my single-node k3s` cluster.
$ curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 664
Note: use --write-kubeconfig-mode 664 to change the permissions for the kubeconfig file to be able to run kubectl without sudo.
--write-kubeconfig-mode 664
kubeconfig
kubectl
sudo
To create an EdgeFaaS pod, we must provide an object specification to Kubernetes. The most common way to deploy Kubernetes objects is to use a .yaml file to send this information to the Kubernetes API via kubectl.
$ cat edgefaas.yaml apiVersion: v1 kind: Pod metadata: name: edgefaas spec: containers: - name: edgefaas image: registry.gitlab.com/arm-research/smarter/edgefaas ports: - containerPort: 8080
Start EdgeFaaS using the edgefaas.yaml
edgefaas.yaml
$ kubectl create -f edgefaas.yaml pod/edgefaas created
If you want to run the EdgeFaaS pod directly, you can run the following command.
$ kubectl run edgefaas --image=registry.gitlab.com/arm-research/smarter/edgefaas --generator=run-pod/v1 pod/edgefaas created
Note: --generator=run-pod/v1 tells Kubernetes to create a pod object using the image.
--generator=run-pod/v1
Retrieve the EdgeFaaS pod IP address and set an environment variable to it.
$ kubectl describe pod/edgefaas | grep IP IP: 10.42.0.19 IPs: IP: 10.42.0.19 $ FAAS=10.42.0.19
$ curl $FAAS:8080/echo -d "Welcome to EdgeFaas on k3s!" Welcome to EdgeFaas on k3s!
One of our goals for the EdgeFaaS was to minimize the footprint of installing new functions. The biggest feature I added to the original watchdog code was enabling the runtime to add and remove functions on the fly, while the functions share a container. In this section, I describe how to manage functions for EdgeFaaS.
Each installed function can be triggered using the function name as a path, edge.faas.ip:8080/function_name.
edge.faas.ip:8080/function_name
EdgeFaaS exposes an HTTP endpoint supporting the following HTTP methods: GET, POST, and DELETE. The management interface can be accessed via the /mgmt path.
/mgmt
The GET method simply lists all the installed functions. Each line prints the function same, which is the same as the path used to trigger the function. In the future, GET could be expanded to print more information about the functions and the runtime.
$ curl edge.faas.ip:8080/mgmt echo func1 func2 ... funcn
Example:
$ curl $FAAS:8080/mgmt echo other_handler
The POST method is used to add a function to the EdgeFaaS container. The POST method accepts an URL pointing to a file containing the function following the specifications described in Function Format. The function file will be downloaded by the runtime, and the name of the function will be the filename with the extension truncated.
$ curl edge.faas.ip:8080/mgmt -d "https://my.site/func/new_func.py" new_func.py has been created
$ curl $FAAS:8080/mgmt -d "https://gitlab.com/arm-research/smarter/edgefaas/edgefaas/-/raw/master/sample_functions/average.py" average.py has been created $ curl $FAAS:8080/average -d "0,1,1" 0.666666666667 $ cat numbers 0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10,0,1,2,3,4,5,6,7,8,9,10 $ curl $FAAS:8080/average -d @numbers 5.0 $ curl $FAAS:8080/mgmt echo average
Installed functions can be overwritten, and this is how functions are updated. The only function that can't be overwritten is echo function since it is considered the default function. echo is used as a quick system check, and some EdgeFaaS functionality depends on it.
The DELETE method removes a function from the runtime. The argument to DELETE is the function name you wish to delete.
$ curl -X "DELETE" edge.faas.ip:8080/mgmt
DELETE in action:
$ curl $FAAS:8080/mgmt echo other_handler $ curl -X "DELETE" $FAAS:8080/mgmt -d "other_handler" $ curl $FAAS:8080/mgmt echo $ curl -X "DELETE" $FAAS:8080/mgmt -d "echo" $ curl $FAAS:8080/mgmt echo
If you try to delete the echo function, the system will not allow it since this is considered the default function.
The initial version of EdgeFaaS supports Python 2.7 functions. Future versions will support other programming languages.
Please keep the following in mind when writing functions for EdgeFaaS:
.py
requirements.txt
We decided to keep the OpenFaaS Python function format. The function file contains a function with the following signature:
def handle(req): <your_code> return req
If you are interested in seeing a valid file, please see the default function, located in the function/handler.py file, but please refrain from modifying this file.
function/handler.py
EdgeFaaS started as an experiment within Arm Research to prototype cloud services deployed on the edge.
Python 2.7.
function/requirements.txt
If you have any questions about EdgeFaaS, please do contact me.
Contact Luis E. Peña
This post is the final blog in a five-part series. Read the other parts of the series using the following links: Part one: SMARTER: A smarter-cni for Kubernetes on the Edge Part two: SMARTER: An Approach to Edge Compute Observability and Performance Monitoring Part three: SMARTER: A Smarter-Device-Manager for Kubernetes on the Edge Part four: SMARTER: Debugging a Remote Edge Device
This post is the final blog in a five-part series. Read the other parts of the series using the following links:
Part one: SMARTER: A smarter-cni for Kubernetes on the Edge
Part two: SMARTER: An Approach to Edge Compute Observability and Performance Monitoring
Part three: SMARTER: A Smarter-Device-Manager for Kubernetes on the Edge
Part four: SMARTER: Debugging a Remote Edge Device