In this post we'll walk through setting up a cluster of Arm machines. We'll use Mesos as the cluster manager, Marathon for container orchestration, Zookeeper as a key-value store, and Weave Net for the container overlay and DNS solution. The high level steps are listed below.
The instructions below were performed on two Softiron Overdrive 1000 computers running Ubuntu. Each machine has Quad-core A-57 AArch64 CPUs. The master node will be called od1000-1, and the agent node will be called od1000-0. A Mesos agent will run on the master node so that tasks can be scheduled on it as well. Running a Mesos agent on a master node can introduce security vulnerabilities on the cluster. It may not be wise to do this on production systems, but for the purpose of this blog, it's ok.
Aside from the Overdrive 1000, Mesos and Marathon have been deployed on other server and networking platforms like the Cavium ThunderX1 (48 AArch64) and Marvell MacchiatoBin (Quad-core A-72 AArch64). The instructions below should work on just about any AArch64 hardware.
Install the following on all nodes in the cluster.
# Update the packages. sudo apt-get update # Install a few utility tools. sudo apt-get install -y tar wget git # Install the latest OpenJDK. sudo apt-get install -y openjdk-8-jdk # Install autotools (Only necessary if building from git repository). sudo apt-get install -y autoconf libtool automake bc # Install other Mesos dependencies. sudo apt-get -y install build-essential python-dev python-six python-virtualenv libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev zlib1g-dev # Install docker sudo apt-get -y install docker.io # Install sbt (complete instructions on sbt website) echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823 sudo apt-get update sudo apt-get -y install sbt # Pre-download Ubuntu Docker image. Will make deployment faster. docker pull ubuntu:16.04
The compilation instructions below is basically what's described in the official Mesos build instructions. The following should be performed on all nodes in the cluster.
Any occurrence of $MESOS_ROOT should be assumed to be a path to the root directory of the Mesos source code.
Clone the Mesos repo and checkout tag 1.4.0. Any recent version of Mesos should be fine too.
# Clone the repo git clone https://git-wip-us.apache.org/repos/asf/mesos.git cd mesos # See list of avialble tags git tag # Checkout 1.4.0 (example) git checkout 1.4.0
Alternatively, rather than cloning the git repo, A tarball of a specific version of the source can be downloaded. All available versions can be found in this Index of /dist/mesos.
With the source downloaded, configure the build environment.
# Change to mesos source root ($MESOS_ROOT). cd mesos # Bootstrap (Only required if building from git repository). ./bootstrap # Configure and build. mkdir build cd build ../configure
The next step is different from the standard instructions posted on the Mesos website. The build will have to be executed twice. The first time around, it will create a number of config.guess files and then fail. The build will fail because these config.guess files aren't compatible with Arm. These files will need to be replaced with Arm compatible versions of the file. Once they are replaced, the build can be executed a second time, which will result in a successful compile.
To understand this further, first run the following command from the Mesos source root.
# Run from Mesos source root ($MESOS_ROOT) cd $MESOS_ROOT find . -name config.guess
There will be a single config.guess file.
Run make from inside the $MESOS_ROOT/build directory created in the previous steps. The build should fail within a few minutes.
# Run from $MESOS_ROOT/build directory. # This will fail the first time it's run. # Optional: Add switch '-jx' where 'x' is the number of cores on the build machine. sudo make
Once it fails, run the find command described above in the source root again. A number of newly created config.guess files will be found.
Next, replace all of these config.guess files with the Arm compatible version. The Arm compatible config.guess file can be found at /usr/share/automake-x.xx/config.guess.
Use this file to replace all of the newly create files. The example commands below show automake version 1.15. Using whatever version is installed in the system should be fine.
# Update all occurrences of config.guess with our installed copy. # This is done from the mesos source root directory. sudo cp /usr/share/automake-1.15/config.guess ./config.guess sudo cp /usr/share/automake-1.15/config.guess ./build/3rdparty/libev-4.22/config.guess sudo cp /usr/share/automake-1.15/config.guess ./build/3rdparty/glog-0.3.3/config.guess sudo cp /usr/share/automake-1.15/config.guess ./build/3rdparty/zookeeper-3.4.8/src/c/config.guess sudo cp /usr/share/automake-1.15/config.guess ./build/3rdparty/protobuf-3.3.0/gmock/build-aux/config.guess sudo cp /usr/share/automake-1.15/config.guess ./build/3rdparty/protobuf-3.3.0/gmock/gtest/build-aux/config.guess sudo cp /usr/share/automake-1.15/config.guess ./build/3rdparty/protobuf-3.3.0/config.guess
After all of the files are replaced, run the make command from above in $MESOS_ROOT/build again.
Once Mesos is compiled, the next step is to install Mesos.
The install command will copy the Mesos binary to /usr/local/bin/. Another important file that gets installed is libmesos. This will be copied to /usr/local/lib. libmesos is loaded by Marathon to interact with Mesos.
The ldconfig command will update the shared object caches so that the run-time linker can find libmesos.
# Install mesos and update shared object cache. # Run this in $MESOS_ROOT/build sudo make install sudo ldconfig
The Mesos source comes with a Zookeeper Java Archive (.jar) already included. Untar Zookeeper and create the config file by making a copy of the sample config that's included. Note the version of Zookeeper provided may be different from what's shown below.
# Example untar of zookeeper 3.4.8 cd $MESOS_ROOT/3rdparty tar xzf zookeeper-3.4.8.tar.gz # Use Zookeeper sample config as our config cd $MESOS_ROOT/3rdparty/zookeeper-3.4.8/conf cp zoo_sample.cfg zoo.cfg
Weave Net will provide the overlay network and DNS solution for the cluster. Weave is multi-platform, so the standard install instructions will work on Arm as well.
# Install Weave sudo curl -L git.io/weave -o /usr/local/bin/weave sudo chmod a+x /usr/local/bin/weave
Instructions to compile Marathon are largely from the Marathon build instructions. The following only needs to be executed on the master node.
Any occurrence of $MARATHON_ROOT should be assumed to be a path to the root directory of the Marathon source code.
Clone the Marathon repo, and checkout version tag v1.5.1. Any recent version of Marathon should be fine too.
# Example clone and compile of v1.5.1 git clone https://github.com/mesosphere/marathon.git cd marathon # See available tags git tag git checkout v1.5.1
Use the machine name, or its IP address for (master_ip).
# Compile Marathon using sbt sbt 'run --master (master_ip):5050 --zk zk://(master_ip):2181/marathon'
Once Marathon is compiled, it will start running automatically. There will be repeating connection failed errors that should look similar to the image below. At this point, kill Marathon with <ctrl-C>.
A single instance of Zookeeper for both Mesos and Marathon will be used. Zookeeper only needs to run on a single node. The master (od1000-1) will be used in this case.
# Start Zookeeper cd $MESOS_ROOT/3rdparty/zookeeper-3.4.8/bin sudo zkServer.sh start
Run the Weave launch command on the master node.
# Launch Weave on master node (od1000-1), and peer with agent node od1000-0 weave launch od1000-0
Weave will need to download some Docker images the first time it's run on a machine. This may take a few minutes.
Now run weave on the agent node.
# Launch weave on agent node (od1000-0) and peer with master (od1000-1) weave launch od1000-1
The launch command can be execute on different nodes in any order.
Note that the peer names should be host names that can be found via DNS. IP addresses can also be used instead of host names. More than one peer connection can be made on a node. This is done by passing the peer names into the weave launch command like below.
# Launch weave net and peer with multiple hosts. weave launch agent0 agent1 ...
To stop Weave Net.
# Will kill the container running a weave net router on the host. weave reset
Start the Mesos master.
(master_ip) is the IP address or host name of the node this command is being run on. In this case it's od1000-1.
(any_directory) should be a path to any empty directory. This directory will contain logs and other information on the Mesos master.
# Start Mesos master. mesos master --ip=(master_ip) --work_dir=(any_directory)
At this point, open a browser and point to machine_name:5050 or ip_address:5050 to see the Mesos UI. This UI shows all of the resources that are available on the cluster.
Next, start the Mesos agents. Run the below on all nodes.
(agent_ip) is the IP address or host name of the node this command is being called on.
(master_ip) is the IP address or host name of the master node (od1000-1) .
(any_directory) should be a path to any directory. This directory will contain logs and other information on the Mesos agent.
# Start Mesos agent so that tasks can be scheduled on this node. sudo mesos agent --ip=(agent_ip) --master=(master_ip):5050 --work_dir=(any_directory) --containerizers=docker,mesos --resources=ports:[1-65535] --docker_socket=/var/run/weave/weave.sock
The resources of this node should appear in the Mesos UI after running the above.
When the agent node is started, there are a few switches that are required.
--containerizers: This switch enables Mesos to translate its configs into Docker CLI options.
--resources: In Mesos, ports are considered resources just like CPUs, RAM, and disk space. When the agent is started, the ports that can be offered as resources must be specified. In the command above, all ports are offered as resources. This is ok for this blog, but offering all ports shouldn't be done in production. Make sure to only offer ports that are actually needed.
--docker_socket: Will tell Mesos to point the Docker client (i.e. Docker CLI command) towards the Weave Net Docker API proxy. Normally the Docker client will talk directly to the Docker Daemon, but since Weave will provide a DNS service, Mesos needs to issue Docker commands to the Weave Net proxy. This proxy allows for containers to get registered with WeaveDNS when they are created. After registration, the Weave Net proxy will forward the Docker command to the Docker daemon. Without this switch, WeaveDNS will not work.
Once the Mesos master and agents have been established, start Marathon.
MESOS_NATIVE_JAVA_LIBRARY: Tells Marathon where to find libmesos. This variable can be placed in ~/.profile to avoid having to set it in the shell that will run Marathon.
(master_ip) is the IP address or host name of the master node.
# Tell marathon where libmesos is export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so # Run marathon cd $MARATHON_ROOT sbt 'run --master (master_ip):5050 --zk zk://(master_ip):2181/marathon'
Open a browser and point it to machine_name:8080 or ip_address:8080 to see the Marathon UI.
To confirm that the cluster has been provisioned properly, start a few containers and verify connectivity on the overlay network.
Create a file and name it basic.json.
{ [1/340] "id": "/arm-demo", "groups": [ { "id": "/arm-demo/group1", "apps": [ { "id": "/arm-demo/group1/ubuntu1", "instances": 2, "cpus": 0.25, "mem": 128.0, "disk": 1000.0, "cmd": "sleep 10000", "container": { "type": "DOCKER", "docker": { "image": "ubuntu:16.04", "network": "BRIDGE", "parameters": [ { "key": "hostname", "value": "ubuntu1.weave.local" } ] } } } ] }, { "id": "/arm-demo/group2", "dependencies": [ "/arm-demo/group1" ], "apps": [ { "id": "/arm-demo/group2/ubuntu2", "instances": 2, "cpus": 0.25, "mem": 128.0, "disk": 1000.0, "cmd": "sleep 10000", "container": { "type": "DOCKER", "docker": { "image": "ubuntu:16.04", "network": "BRIDGE", "parameters": [ { "key": "hostname", "value": "ubuntu2.weave.local" } ] } } } ] } ] }
Some detail on this JSON file:
The JSON file describes two Marathon application groups (group1 & group2). Each group will run a single Marathon application. Each application will run two instances of an Ubuntu container that calls sleep. Each container instance uses .25 CPUs, 128MB of RAM, and 1GB of disk space. Thus, to run the 4 containers described in the JSON file, Mesos will require that 1 CPU, 512MB of RAM, and 4GB of disk space be available on the cluster. Mesos also requires that the requested resources of a single container instance (.25 CPUs, 128MB RAM, and 1GB Storage) must all be available on a single host for it to deploy on that host.
To support the Weave Net overlay, notice that a hostname parameter is set with the domain .weave.local. This is required so that WeaveDNS registers the host name. Without this, the containers cannot discover each other via host/service name.
Last, the sequence in which the containers startup is controlled by the group dependencies. In the example file above, /arm-demo/group2 is dependent on /arm-demo/group1. When this file is posted to Marathon, the containers in group1 will start up before the containers in group2.
The Marathon HTTP API is used to request Mesos to startup the containers.
(master_ip) is the IP address or host name of the node (od1000-1) running Marathon.
# Issue HTTP post to Marathon to start services in basic.json. curl -X POST http://(master_ip):8080/v2/groups -d @basic.json -H "Content-type: application/json"
After running this command, take a look at the Marathon UI. After a minute or two, the containers should show as running. The containers should startup quickly because their base image was downloaded in the prerequisites section of this post.
Open the Mesos UI. It should also show that the four containers are running.
Run docker ps on each node to see the containers.
OD1000-1 (master):
OD1000-0:
There are three containers running on each node. Two Ubuntu 16.04 containers and a Weave Net Router.
Notice that the container instances in each group have split across the two nodes of the cluster. That is, there's one ubuntu1 (group1) container on od1000-0, and the other is on od1000-1. The same is true for the two container instances of application group2. This is desirable, as it provides fault tolerance for each group.
Confirm that WeaveDNS has registered all four containers.
All four containers have registered with WeaveDNS. The containers with IP addresses of 10.40.x.x are on one node, while the containers with IP addresses of 10.32.x.x are on the other. The subnet for this overlay is 10.32.0.0/12, so these containers are all on the same VXLAN subnet as expected.
So far, it looks like there should be connectivity between containers. To be sure, confirm connectivity from the perspective of one of the containers.
Shell into a container on node od1000-1.
(container_hash) is the hash value of the container to shell into. It can be obtained by running the docker ps command.
# Shell into container running on this host. docker exec -ti (container_hash) bash
Once there is a shell in the container, ping and nslookup have to be installed.
# From container shell, install nslookup and ping. apt update && apt install -y dnsutils iputils-ping
From within this ubuntu2 instance, check DNS by running nslookup against host names ubuntu1 and ubuntu2.
There are two IP address entries for each host name as expected.
If nslookup is run more than once with the same host name, the order of reported IP addresses will change. This is the built in load balancing feature of WeaveDNS.
Ping all IP addresses from this ubuntu2 instance to confirm connectivity to all other containers.
Given the above, connectivity and DNS is confirmed across the overlay network.
We've proved that Mesos, Marathon, Zookeeper, and Weave Net are all functioning properly in an Arm cluster. With the exception of Weave Net, these are all components of the Datacenter Operating System (DC/OS) open source project. Given the above results, it's very likely that DC/OS will run on Arm without issues as well. Perhaps exploring DC/OS will be the subject of a future blog post. Last, aside from the special handling of the config.guess files when compiling Mesos, all of the instructions presented above are architecture agnostic. This is what's desired for a heterogeneous data center.
We encourage readers to get involved with the Works on Arm project. This project aims to further develop the Arm Software ecosystem in the data center. The idea of the Works on Arm project is to take Open Source projects, deploy them on Arm platforms, and debug any functional and performance issues found in the process. Much like what we've done above with Mesos/Marathon, all it takes to get started is to build and run the SW as intended by its developers.
[CTAToken URL = "https://www.worksonarm.com/" target="_blank" text="Works on Arm Project" class ="green"]