In this blog, we explore a data collection, processing, and analysis use case that spans edge to cloud with the Splunk Universal Forwarder on an Arm-based device.
Edge devices typically include various of sensors in a compact form factor that generates data. This data is used to make real-time decisions. Once that real-time decision is made, the data sets are not discarded but are rather retained, aggregated, and used to research historical patterns and further build new predictive models. To process this bulk data, we need intelligent computing infrastructure in all locations – edge and cloud. The Edge requires power efficiency, and the Cloud requires cost-performance efficiency. Arm is a natural fit in both scenarios. Let us dig into the details now.
For this Edge to Cloud use case, the cloud side consists of Splunk Enterprise running in an AWS EC2 instance. This instance is used to index and visualize data that is coming in from the edge. The edge side of the use case runs the Splunk Universal Forwarder on an NVIDIA Jetson Xavier NX platform based on the Arm architecture. The Jetson device has an array of sensors to sample the conditions of the environment where it is located. It then uses the Splunk Universal Forwarder to send the sensor data to Splunk Enterprise in the cloud.
Splunk helps organizations around the world turn data into doing. Splunk technology is designed to investigate, monitor, analyze, and act on data at any scale.
For this use case, we will be using the Splunk Enterprise and Splunk Universal Forwarder products.
Splunk Enterprise is a software product that enables you to search, analyze, and visualize the data gathered from the components of your IT infrastructure or business. Splunk Enterprise takes in data from websites, applications, sensors, devices. After you define the data source, Splunk Enterprise indexes the data stream and parses it into a series of individual events that you can view and search.
The Splunk universal forwarder collects data from a data source or another forwarder and sends it to a forwarder or a Splunk deployment. With a universal forwarder, you can send data to Splunk Enterprise, Splunk Light, or Splunk Cloud. It also replaces the Splunk Enterprise light forwarder. The universal forwarder is available as a separate installation package.
The following is a top-level image of the use case setup.
On the left, we have the AWS cloud hosted part of this use case. It is made up of a VPC and subnet hosted in the us-east-1 region. Within the subnet, we deploy an M6i instance that is running Ubuntu which has Splunk Enterprise installed. The instance is connected to the public Internet through an Internet gateway and routing table that connects the instance to the public internet. On the right, we have the edge hosted part of this use case. This includes the internet and a Jetson Xavier NX. The Jetson board has sensors attached to it which sample temperature, humidity, pressure, VOC gasses, and light intensity. The Jetson board also has the Splunk Universal Forwarder installed. The forwarder is configured to read samples of the sensor data and send it to the Splunk Enterprise instance in the cloud for indexing. Once Splunk Enterprise has indexed the data, we can search and visualize the sensor data. The following is a partial image of the dashboard showing the pressure data generated by the edge device.
All the files required to deploy this use case are hosted on github. Within the repo, there are various README.md files that further explain how to deploy the demo including which commands to run. We encourage that time is taken to read through those files before attempting to deploy. In the following section, we refer to the root of the repo as $REPO_ROOT.
The tools required to deploy the demo are Terraform and Ansible. Terraform is used to deploy the cloud side infrastructure (EC2 instance, gateway, VPC). Ansible is used to install Splunk Enterprise in the cloud and to install the Splunk Universal Forwarder on the edge. The edge device does not have to be a Jetson platform, it can be any Arm-based device. The sensors we used on the Jetson board are documented in the READMEs in the repo. However, it is not a requirement to have these sensors to deploy the demo. It is possible to create a Python based mock data generator to remove the requirement of acquiring sensors. This mock data approach is also documented in the repo.
As noted above, the cloud side components are deployed automatically with the terraform configuration files (see $REPO_ROOT/enterprise/terraform). The variables.tf file can be used to change things like AMI, instance type, region, and availability zone. If changing the AMI, note that this demo has only been tested with Ubuntu images. The most important variables to set are aws_key and ingress_ips. aws_key is the name of the RSA/EdDSA key pair that is used to access the instance (with SSH). ingress_ips is a list of IP addresses that are allowed to access the instance. Setting this variable allows for accessing the UI, data forwarding between the edge device and the cloud, and to pull down edge device configuration updates. The security_groups.tf file is where this list of IP addresses is applied. The default value for this variable is blank. This means the user has to explicitly list the IP addresses that are allowed to access the Splunk Enterprise installation. If nothing is specified, an error occurs. The preferred list of IPs to use are the public IP address of the edge device, and the public IP address of the laptop used to run terraform and ansible. Instructions on how to set these variables is in the READMEs.
###################################################################### # Variables related to general setup of the infrastructure ###################################################################### variable "aws_region" { description = "The AWS region we want to deploy on." type = string default = "us-east-1" } ################################### # Variable related to instance deployment ################################### variable "enterprise_instance_ami" { description = "The AMI to use for enterprise_instance_type" type = string default = "ami-09e67e426f25ce0d7" } variable "enterprise_instance_type" { description = "The EC2 instance type to use for splunk enterprise" type = string default = "m6i.large" } variable "instance_az" { description = "Availability Zone we want for splunk" type = string default = "us-east-1a" } variable "ingress_ips" { description = "A list of IP address we will allow to connect to the splunk instance(s), 0.0.0.0/0 is all ips. all ips is ok for debug, but don't use it for demos" type = list(string) default = [""] } variable "aws_key" { description = "The AWS public key to push into the instances for SSH/SCP." type = string default = "" }
The installation and configuration of Splunk Enterprise is done using the ansible files included under the $REPO_ROOT/enterprise/ansible directory. There are two things to be mindful of. The first is the name of the Splunk Enterprise installer file and its location. The installer is not included in the repo. This has to be downloaded into $REPO_ROOT/enterprise/installers from the Splunk website. The installers directory must be created if it does not exist. The second item is that TLS certificates need to be created and placed in the $REPO_ROOT/enterprise/server_certs directory. The READMEs and Splunk documentation explain how to do this. Last, when we run the ansible playbook, we are asked to set an admin password. Remember this password as it is needed to log into Splunk Web. More details are in the README file.
--- - name: Install & Setup Splunk vars: ent_user: "ubuntu" remote_home: "/home/{{ ent_user }}" splunk_ent_path: "/opt/splunk" splunk_ent_local: "{{ splunk_ent_path }}/etc/system/local/" splunk_ent_certs: "{{ splunk_ent_path }}/etc/auth/mycerts/" splunk_ent_app: "{{ splunk_ent_path }}/etc/apps" splunk_ent_deployment_apps: "{{ splunk_ent_path }}/etc/deployment-apps/" splunk_deb: "splunk-8.2.1-ddff1c41e5cf-linux-2.6-amd64.deb" vars_prompt: - name: admin_password prompt: Enter Splunk Enterprise Admin Password private: yes confirm: yes hosts: all remote_user: ubuntu become: true become_method: sudo tasks: - name: Copy splunk installer and various config files copy: src: "{{ item.src }}" dest: "{{ item.dest }}" with_items: - { src: "../installers/{{ splunk_deb }}", dest: "{{ remote_home }}" } - { src: ../server_certs/, dest: "{{ splunk_ent_certs }}" } - { src: ../splunk_config/web.conf, dest: "{{ splunk_ent_local }}" } - { src: ../splunk_user/user-seed.conf, dest: "{{ splunk_ent_local }}" } - { src: ../splunk_inputs/inputs.conf, dest: "{{ splunk_ent_local }}" } - { src: ../splunk_apps/edge_sensor_demo, dest: "{{ splunk_ent_app }}" } - { src: ../splunk_deployment_apps/edge_array_output, dest: "{{ splunk_ent_deployment_apps }}" } - { src: ../splunk_deployment_apps/edge_array_input, dest: "{{ splunk_ent_deployment_apps }}" } - name: Install the splunk package apt: "deb={{ remote_home }}/{{ splunk_deb }}" - name: Seed the password into the user-seed.conf file on the Splunk Enterprise host replace: path: "{{ splunk_ent_local }}/user-seed.conf" regexp: '<PW>' replace: "{{ admin_password }}" - name: Add splunk enterprise IP to conf files replace: path: "{{ item.path }}" regexp: 'splunk_ent_ip' replace: "{{ lookup('file', './host') }}" with_items: - { path: "{{ splunk_ent_deployment_apps }}/edge_array_output/default/outputs.conf" } - name: Start Splunk Enterprise command: "/opt/splunk/bin/splunk --accept-license restart splunkd"
The installation and configuration of the Splunk Universal Forwarder on the edge device is done using the ansible files included under the $REPO_ROOT/fwd/ansible directory. In this ansible playbook, we must be mindful of three things. The first is the installer file and its location. The installer is not included in the repo. This has to be downloaded into $REPO_ROOT/fwd/installers from the Splunk website. The installers directory must be created if it does not already exist. The second is that an ansible inventory file needs to be created in the $REPO_ROOT/fwd/ansible/hosts directory. This file must contain the IP address of the edge device. The last thing to be mindful of is that TLS certificates must be copied into the $REPO_ROOT/enterprise/indexer_certs directory. More details are in the README file.
--- - name: Install & Setup Splunk forwarder vars: splunk_fwd_path: /opt/splunkforwarder splunk_fwd_local: "{{ splunk_fwd_path }}/etc/system/local/" splunk_fwd_certs: "{{ splunk_fwd_path }}/etc/auth/mycerts" splunk_tgz: "splunkforwarder-8.2.1-ddff1c41e5cf-Linux-armv8.tgz" python_scripts_path: "/home/{{ fwd_user }}" vars_prompt: - name: fwd_password prompt: Enter Splunk Forwarder Admin Password private: yes confirm: yes - name: fwd_user prompt: Enter Splunk Forwarder User private: no hosts: all remote_user: "{{ fwd_user }}" become: true become_method: sudo tasks: - name: Stop the Splunk Forwarder if it's running command: "/opt/splunkforwarder/bin/splunk stop --accept-license" ignore_errors: yes - name: Delete splunk forwarder if it exists file: path: "{{ splunk_fwd_path }}" state: absent - name: Copy and install the forwarder & sensor interface scripts unarchive: src: "../installers/{{ splunk_tgz }}" dest: /opt - name: Copy splunk forwader configs & certs. Also copying sensor interface scripts. copy: src: "{{ item.src }}" dest: "{{ item.dest }}" with_items: - { src: ../local/, dest: "{{ splunk_fwd_local }}" } - { src: ../indexer_certs/, dest: "{{ splunk_fwd_certs }}" } - { src: ../../python, dest: "{{ python_scripts_path }}" } - name: Make python sensor script executable file: path: "{{ python_scripts_path }}/python/read_sensors.py" mode: 'u+x,g+x,o+x' - name: Add splunk enterprise IP to conf files replace: path: "{{ item.path }}" regexp: 'splunk_ent_ip' replace: "{{ lookup('file', './splunk_ent_ip') }}" with_items: # - { path: "{{ splunk_fwd_local }}outputs.conf" } - { path: "{{ splunk_fwd_local }}deploymentclient.conf" } - name: Seed the password into the user-seed.conf file on the Splunk Enterprise host replace: path: "{{ splunk_fwd_local }}/user-seed.conf" regexp: '<PW>' replace: "{{ fwd_password }}" - name: Start Splunk Forwarder command: "/opt/splunkforwarder/bin/splunk start --accept-license"
At this point, everything is up and running and we can connect and sign into Splunk Web.
The one difference we see from the default Splunk Web UI is the addition of the Edge Sensor Demo application. It appears towards the top left of the screen. Before we can open the dashboard to see the live sensor data, we must setup a server class. This lets the edge device understand what configuration it must pull down. Once the edge device has downloaded this configuration, it starts forwarding data to the indexer. The server class setup is documented in the Splunk Enterprise documentation. The applications that must be added to the server class are edge_array_input and edge_array_output. The client name for the edge device is EdgeArray-00. This is evident when clicking through the server class creation setup menus. When the server class is created, and the configuration is downloaded by the edge device. The Forwarder Management screen should show that two files have been downloaded by the edge device. More details are in the README file.
At this point, we can go back to the main Splunk Web screen, click the Edge Sensor Demo Application, and then click the Edge Sensor Array dashboard. This brings up various graphs showing the live edge sensor readings. With everything up and running, we encourage users to explore the configuration files in both Splunk Enterprise and the Universal Forward. We also encourage users to explore the Splunk Enterprise and Splunk Universal Forwarder documentation. Last, we would like to encourage users to try deploying the Splunk Universal Forwarder on their Arm-based edge devices with their own use cases.
In summary, we have described and provided collateral (on github) that demonstrate how to deploy the Splunk Universal Forwarder to an Arm-based device. And to configure it to communicate with the Splunk Enterprise on an AWS EC2 instance. The Splunk universal forwarder can then collect data from multiple sensors at the edge and send it to the Splunk Enterprise. Splunk Enterprise can be used to search, analyze, and visualize this data gathered at the edge.
For any queries related to your software workloads running on Arm Neoverse platforms, feel free to reach out to us at sw-ecosystem@arm.com.