In this post, we explore running Edge Delta on an Arm-based Jetson Xavier NX at the edge. We demonstrate how Edge Delta can be used to analyze data on an edge device to minimize the amount of data sent to the cloud. We also demonstrate how to extract metrics from the edge device and send them to the Edge Delta admin GUI for visualization.
Edge Delta gives enterprises distributed stream processing, federated machine learning, and automated observability data pipeline capabilities. This empowers DevOps, Security, and SRE teams to consume and automatically route all forms of optimized machine data output (logs, metrics, traces). And it provides them with the scalability and efficiency to eliminate technical and financial limitations. This results in organizations that can identify and remediate operational and security incidents more accurately, orders of magnitude faster, and more securely than previously possible with traditional centralized systems.
On the edge side of this demo, we have the Jetson board which runs an Ubuntu 18.04 image with the Jetpack SDK installed. The Jetpack SDK allows easier use of the CUDA and Tensor cores that are in the Jetson GPU. We use these GPUs to accelerate Machine Learning (ML) inference. To do the inference, we installed the Nvidia Triton Inference server on the Jetson. Triton is a tool that hosts ML models for clients to use. A client (local or remote) can submit inference requests to Triton using HTTP or gRPC (we used gRPC). The request includes data like which ML model to use along with the required input tensors. Triton processes the request with GPU acceleration and sends the results back to the client. In our case, we used Triton to run an object detection model on video frames captured from an attached camera. The model we used was the ssd_mobilenet_v2_2 model that can be found on Tensor Flow Hub. This model can detect many everyday objects within video frames. To keep this post simple, our object detection client code limited its reporting to the number of people the camera sees within it's field of view. Aside from the camera, we also have an integrated temperature/pressure/humidity sensor attached to the Jetson. To keep this post simple, we only use the temperature data from this sensor.
We used the Arm Research group's work on the SMARTER project as the basis for the object detection code in this demo. More specifically, we borrowed and modified the image-detector code from the SMARTER project repo. It is not necessary to modify this code to get it to work with Edge Delta, but we modified it since we use this code in other experiments. The main difference in our version of this code is the removal of some unneeded features and some refactoring. And we switched out the object detection ML model with the ssd_mobilenet_v2_2 model as noted earlier. We would like to encourage readers to explore the SMARTER project as it is a very complete example of an edge and cloud use case.
ssd_mobilenet_v2_2
The cloud side of the use case is straight forward. Edge Delta is a hosted service so all that is needed is an account. Free trials are offered which is helpful for experimentation and exploration purposes. Once an account is created, you can sign in to the Admin GUI. From this GUI, you can obtain instructions and commands on installing the Edge Delta agent on edge devices. In our case, we need only a single command to install the agent on the Jetson. At this point, we are ready to configure the Edge Delta agent, but first we must understand what data we would like to analyze.
Edge Delta supports various types of input data sources. In this demo we use two types of input data sources, files and execs (scripts). Overall, the demo uses four data sources in all.
/var/log/object_data
/var/log/sensor_data
/var/log/auth.log
The admin GUI also provides information on how much data has been analyzed at the edge as well as streamed to other services. In our case, we did not integrate Edge Delta with other services and tools to keep things simple (no destinations to stream data to in the following image). We recommend reviewing the documentation on integrations to learn more about this feature.
We do not go into detail on all Edge Delta configuration options, only what is needed to understand how this demo was deployed. For more information on configuring an Edge Delta agent, we suggest reviewing their documentation.
In the configuration file, we define input data sources and processors. The processors can be used to manipulate and extract data from the inputs. Also, processors can search for patterns and anomalies in the data and alert admins when anomalies are found. Edge Delta's ability to process and analyze data locally without having to send and store it all in the cloud is one of its key benefits. The complete configuration we used follows; it is separated into its logical components so that the configurations can be described more easily.
Agent settings are for the agent that is running on the Jetson.
version: v2 agent_settings: tag: arm_jetson_demo log: level: info anomaly_capture_size: 1000 anomaly_confidence_period: 30m
This is mostly default; we just changed the tag to arm_jetson_demo.
arm_jetson_demo
Edge Delta allows for the definition of filters. One of the uses of filters is to remove extraneous information from input data to ease the job of processors. The following code filter removes input data lines that start with a hashtag. The name of the filter is clean_triton_log. We reference this filter further down in the configuration.
clean_triton_log
filters: - name: clean_triton_log type: regex pattern: "^#.*" negate: true
Below we select the input data sources.
inputs: execs: - labels: "triton_metrics" command: "curl localhost:8002/metrics" interval: 1m filters: - clean_triton_log files: - labels: "system_logs, auth" path: "/var/log/auth.log" - labels: "env_sensors" path: "/var/log/sensor_data" - labels: "obj_detections" path: "/var/log/object_data"
The first data source we see is triton_metrics. To obtain the Triton metrics, we used an executable (execs) input type. This allows us to run the curl command on localhost port 8002. We use localhost because this configuration is for the agent that is running locally on the Jetson alongside the Triton server. Notice that for the execs, we apply the filter called clean_triton_log that we described earlier. This removes lines that start with a hashtag which we know contains no useful data. The last input sources are the log files we described earlier. Note that env_sensors is the label for the temperature sensor input data, we reference this further down in the configuration.
triton_metrics
curl
env_sensors
The following is our processor configuration. These processors are what tell Edge Delta what to do with input data.
processors: cluster: name: clustering num_of_clusters: 25 samples_per_cluster: 2 reporting_frequency: 1m regexes: - name: "temperature" pattern: "\\s+temperature=(?P<F>\\d*\\W?\\d+)" trigger_thresholds: anomaly_probability_percentage: 95 - name: "cam" pattern: ",\\s+(?P<detected>person)=(?P<count>\\d+)" dimensions: ["detected"] trigger_thresholds: anomaly_probability_percentage: 95 - name: "ssd_mobilenet_v2_2" pattern: "^nv_inference_(?P<metric>.*){model=\"ssd_mobilenet_v2_2\",version=\"1\"}\\W+(?P<value>\\d*\\.?\\d+)" dimensions: ["metric"] trigger_thresholds: anomaly_probability_percentage: 90 - name: "auth_failed" pattern: "\\b(?:[Aa]uthentication failure|FAILED SU|input_userauth_request: invalid user|Invalid user|Failed publickey|Failed password)\\b" trigger_thresholds: anomaly_probability_percentage: 95 - name: "auth_success" pattern: "\\b(?:su:|sudo:|sshd:|sshd\\[|pam_unix).*(?:\\b[Aa]ccepted|session opened|to\\b.*\\bon)\\b" trigger_thresholds: anomaly_probability_percentage: 95 - name: "auth_root" pattern: "\\b(?:sudo|root|su)\\b" trigger_thresholds: anomaly_probability_percentage: 95 - name: "auth_su_attempt" pattern: "\\b(?:su:|su\\[).*(?:[Aa]uthentication failure|FAILED SU|input_userauth_request: invalid user|Invalid user|Failed publickey|Failed password)\\b" trigger_thresholds: anomaly_probability_percentage: 95
At the top, we see a cluster processor defined. These are used to find patterns in the input data and to cluster them together. Having this defined here does not mean that clustering will happen. It still must be "called" in a workflow which we will see in the next section. Below the cluster section, we see various regexes processors. These tell Edge Delta how to extract the metrics we care about. We can see regexes for extracting temperature, person counts from the camera (called cam), and Triton metrics (called ssd_mobilenet_v2_2) related to the ssd_mobilenet_v2_2 model. Like the cluster processor, having these regexes defined here does not mean they are being used. We have to "call" them in a workflow.
cluster
regexes
Workflows must be defined to tie together the various inputs and processors. The following is what those workflows look like for our demo.
workflows: edge_env_sensors_workflow: input_labels: - env_sensors processors: - clustering - temperature edge_camera_workflow: input_labels: - obj_detections processors: - clustering - cam edge_triton_workflow: input_labels: - triton_metrics processors: - clustering - ssd_mobilenet_v2_2 stats_workflow: input_labels: - system_stats example_workflow: input_labels: - system_logs processors: - clustering - auth_failed - auth_success - auth_root - auth_su_attempt
Here we see a few different workflows defined. The first is the edge_env_sensors_workflow. The label env_sensors tells the agent that /var/log/sensor_data is used as the data input for this workflow. Just under the input_labels section, we see the processors. Here we "call" the clustering and temperature processors. This tells Edge Delta to search the /var/log/sensor_data file for patterns and anomalies with the clustering processor, and to extract the temperature readings using the temperature processor. We leave the other workflows up to the reader to study.
edge_env_sensors_workflow
input_labels
clustering
temperature
Next, we explore some of the Edge Delta admin GUI while the demo is deployed. We see that pattern and anomaly detection is a key and useful feature of Edge Delta.
The following is an image of the patterns section in the Admin GUI. This is generated for workflows which use the clustering processor.
The Negative Patterns graphs show us two interesting things. The first is that it noticed multiple failed SSH logins into the Jetson at 13:43. This is a useful piece of information as it could indicate a security issue. When these failed attempts occurred, we also received an email alerting us about these failed login attempts. That said, we know this is not an issue because we generated this intentionally by purposely entering an invalid password multiple times while trying to SSH into the Jetson. The second thing we notice from the graph is that we consistently see inference request failures coming from the Triton server logs (logs that start with nv_inference_request). What is interesting is that it is detecting these failures on models that are loaded into the Triton server, but not actually being used. For example, vggish is a model that is loaded, but not being used as part of this demo. This is something that would have gone unnoticed had we not configured Edge Delta to check the Triton logs for patterns with the cluster processor. Now we know there might be a Triton server configuration issue that must be investigated. There are other graphs and logs in the patterns section of the admin GUI, but we leave this up to readers to explore further.
vggish
The anomalies section contains a few different graphs. We show one of them in the interest of brevity.
In the previous graph, the outer rings are various events and metrics the agent analyzes. On the left, we see one of those events highlighted in red. This is hard to read, but it says "auth_failed" which are the failed SSH connection attempts we saw in the Negative Patterns section. Edge Delta is (rightly) labeling these failed login attempts as an anomaly that ought to be investigated.
The metrics section is straight forward. The metrics shown here are extracted with the regexes processors in our configuration file. The following are temperature and person count graphs.
As we can see, the temperature where the Jetson is located is stable and consistent. As for the person detection graph, we see that the number of people seen by the camera starts at 0 and eventually rises to 3.
Overall, it was very easy to get up and running with Edge Delta. In fact, we spent more time developing the camera/sensor interface code than we did setting up Edge Delta itself. We would like to encourage readers to give Edge Delta a try on Arm based edge devices. Especially for use cases where edge bandwidth might be limited or where there is cost/performance sensitivity to storing and searching data in the cloud.
Try Edge Delta on Arm