In this post, we'll discuss Image Manifests, Manifest Lists, and their role in enabling multi-architecture Docker registries. Then we'll walk through an example of setting up a multi-architecture Docker registry on a MacchiatoBin. Although we are using the MacchiatoBin in this blog, the instructions presented below will work on just about any AArch64 HW available on the market. The setup of a MacchiatoBin was discussed in Configuring The MacchiatoBin For Kubernetes and Swarm.
An Image Manifest is a file that contains information about the composition of a particular image. Details on Image Manifests can be found on the Image Manifest V2 description page. An example Image Manifest is shown below.
{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 7023, "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 32654, "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 16724, "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 73109, "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736" } ] }
The key portion of this file is the section labeled layers. It lists all the layers that make up the image. The first layer is the base image, and the subsequent layers are the additional layers needed to compose the final image. This file is automatically created in the registry when an image is pushed into it.
layers
A Manifest List is a file that contains a list of Image Manifests. Below is the Manifest List for the Docker Hub image ubuntu:16.04.
ubuntu:16.04
{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1357, "digest": "sha256:9b47044b1e79b965a8e1653e7f9c04b5f63e00b9161bedd5baef69bb8b4c4834", "platform": { "architecture": "amd64", "os": "linux" } }, { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1357, "digest": "sha256:6741f93913a11e1be518f95acd00ef35b3408e12a9a39db4ffe757a9cec94f1b", "platform": { "architecture": "arm", "os": "linux", "variant": "v7" } }, { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1357, "digest": "sha256:8aecae775e1f81d3889929ef15647187b414c833b0798d060bfd778bee668ced", "platform": { "architecture": "arm64", "os": "linux", "variant": "v8" } }, { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1357, "digest": "sha256:14aa215f96eb6996a323f4290aea3b935dd69636ef8f04e82812ac2bbe277bb9", "platform": { "architecture": "ppc64le", "os": "linux" } }, { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1357, "digest": "sha256:55f36465901b2f01fcdceecb69d0bc6aacd1c344d01ae25fba6e001f9f7b2dfb", "platform": { "architecture": "s390x", "os": "linux" } } ] }
The key portion of this file is the section labeled manifests. This section lists six Image Manifests. The main difference between the listed Image Manifests is the architecture. On closer inspection, we can see that two of these Image Manifests are for Arm Ubuntu images.
manifests
The Manifest List is how Docker Hub supports different architectures with the single image name and tag of ubuntu:16.04. When a client requests to pull ubuntu:16.04, the registry will check the Manifest List to see if there is an image that matches the client's architecture. If there is, the registry will serve that version of the image to the client.
Let's walk through setting up a registry on the MacchiatoBin, then pull images from the registry from both the MacchiatoBin and an x86 machine. This will verify that the registry can serve multi-architecture images.
Install the latest stable version of Docker. This can be done by running the following command on both the MacchiatoBin and on the x86 machine.
curl -sSL https://get.docker.com | sh
Check the version that was installed.
marvell@macchiato-1:~$ docker version Client: Version: 18.06.1-ce API version: 1.38 Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:32:42 2018 OS/Arch: linux/arm64 Experimental: false Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:30:47 2018 OS/Arch: linux/arm64 Experimental: false
Notice that both the client and server have their experimental flags set to false. We will need the client's (Docker CLI) experimental flag set to true to use the docker manifest commands. The docker manifest commands are used to create and push the Manifest List. We will also need the server's (Docker Daemon) experimental flag set to true. This will allow the use of the --platform flag with the docker pull command (explained later).
docker manifest
--platform
docker pull
Before we turn on the experimental features, let's try the docker manifest and docker pull commands to verify that we need the experimental features enabled.
marvell@macchiato-1:~$ docker manifest docker manifest is only supported on a Docker cli with experimental cli features enabled marvell@macchiato-1:~$ docker pull --platform arm64 ubuntu:17.04 "--platform" is only supported on a Docker daemon with experimental features enabled
As expected, we need to enable experimental features.
To turn on the client experimental features. Add the below to the file ~/.docker/config.json.
~/.docker/config.json
{ "experimental": "enabled" }
To turn on the daemon experimental features. Add the below to the file /etc/docker/daemon.json.
/etc/docker/daemon.json
{ "experimental": true }
Notice that the value for the experimental key is different in each case.
For the changes to take effect on the Docker daemon, we need to restart it.
sudo systemctl daemon-reload sudo service docker restart
If we check the Docker version again, we'll see that the experimental features have been enabled.
marvell@macchiato-1:~$ docker version Client: Version: 18.06.1-ce API version: 1.38 Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:32:42 2018 OS/Arch: linux/arm64 Experimental: true Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:30:47 2018 OS/Arch: linux/arm64 Experimental: true
The current Docker registry repo builds for x86 only. This is because the repo contains an x86 pre-built binary that gets copied into the image. As I've mentioned in previous blogs (Architecture Agnostic Container Build Systems and Cloud Management Tools on Arm), this is a practice that should be avoided. Fortunately, there is a PR which addresses this issue. We will use this PR to build a Docker registry image for AArch64.
Start by cloning the registry image repo.
git clone https://github.com/docker/distribution-library-image.git cd distribution-library-image
Next, we'll fetch the PR, and checkout the branch that is created after the fetch.
git fetch origin pull/74/head:multiarch git checkout multiarch
Now run the update.sh script. We probably don't have to run it, but it won't hurt. The script just ensures that you will build the latest registry image. I think this script might just be a utility for the project maintainers.
update.sh
marvell@macchiato-1:~/distribution-library-image$ ./update.sh 2.6 Updating distribution 2.6... Full version: 2.6.2 Done.
As shown above, the version we will build is 2.6.2 which is the latest release of the registry.
Run the docker build command to build the registry.
docker build
marvell@macchiato-1:~/distribution-library-image$ docker build -t registry:2.6.2 ./2.6/ marvell@macchiato-1:~/distribution-library-image$ docker images | grep registry registry 2.6.2 7602ea1af04f 31 seconds ago 30.7MB
Finally, to run the registry, we'll follow the deploy a registry server instructions. To keep things simple, we will run the registry in an insecure manner. That is, we will not use TLS to secure communication with the registry.
marvell@macchiato-1:~/distribution-library-image$ docker run -d -p 5000:5000 --restart=always --name registry registry:2.6.2 marvell@macchiato-1:~/distribution-library-image$ cd ~ marvell@macchiato-1:~$
You can run docker ps to verify that the container is running.
docker ps
Now that the registry is running, we'll need to push AArch64 and x86 images into the registry. Rather than creating a Dockerfile to build the images, we'll take a shortcut. We'll download the AArch64 and x86 Ubuntu images from Docker Hub, retag them, and then push them into our private registry.
Download the AArch64 image, retag it, and push it to the registry.
marvell@macchiato-1:~$ docker pull ubuntu:17.04 marvell@macchiato-1:~$ docker tag ubuntu:17.04 macchiato-1:5000/my-ubuntu-arm64:17.04 marvell@macchiato-1:~$ docker rmi ubuntu:17.04 marvell@macchiato-1:~$ docker push macchiato-1:5000/my-ubuntu-arm64:17.04 ... 17.04: digest: sha256:ea65878d686232eb82eaaffd68676c7334af43446e0005d0c2810f34c9f9d376 size: 1357
Download the x86 image, retag it, and push it to the registry. Notice that we use the --platform flag. This informs the registry (Docker Hub) that we want an x86 (amd64) image even though our client is AArch64.
marvell@macchiato-1:~$ docker pull --platform amd64 ubuntu:17.04 marvell@macchiato-1:~$ docker tag ubuntu:17.04 macchiato-1:5000/my-ubuntu-amd64:17.04 marvell@macchiato-1:~$ docker rmi ubuntu:17.04 marvell@macchiato-1:~$ docker push macchiato-1:5000/my-ubuntu-amd64:17.04 marvell@macchiato-1:~$ docker rmi macchiato-1:5000/my-ubuntu-amd64:17.04 ... 17.04: digest: sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459 size: 1357
When we tagged the images, noticed that we prefixed the image name with the machine name that is hosting our registry. This will tell Docker that the image belongs to a private registry. In this case, it's a private registry hosted on port 5000 on macchiato-1. Also, recall that when the image is pushed to the registry, an Image Manifest is automatically created on the registry. We'll explore this a little later.
Last, remove the local copies of the images that were just pushed. This ensures we are only interacting with the images that are hosted on our registry.
marvell@macchiato-1:~$ docker rmi macchiato-1:5000/my-ubuntu-arm64:17.04 marvell@macchiato-1:~$ docker rmi macchiato-1:5000/my-ubuntu-amd64:17.04 marvell@macchiato-1:~$ docker system prune
Before we create the Manifest List, let's take a look at the Image Manifests for the two images we just pushed. To do this, we'll use the docker manifest inspect command. We will also need to add the --insecure flag to the command. This is because the docker manifest inspect command only communicates with the Docker registry. It does not query the client's Docker daemon to figure out which insecure registries are allowed.
docker manifest inspect
--insecure
View the arm64 Image Manifest.
marvell@macchiato-1:~$ docker manifest inspect --insecure macchiato-1:5000/my-ubuntu-arm64:17.04 { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 3614, "digest": "sha256:96ad946778d8c777aeb78278e22fb0c1c6de37315e6bfce21654210dc648d789" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 35796134, "digest": "sha256:dec7072e8ec5718e527d5c9de1313867b72446297190172e1e3d9de0c5b830ee" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 854, "digest": "sha256:a52218c98a190f0cb50bd8f73946e55694027b173c9a6efcb59cfb76b4e3668c" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 539, "digest": "sha256:34ce812f23df9ecae2602041e53b5b15980e5521f521ac7a4833f848854d03ff" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 857, "digest": "sha256:9142cb3047dbb70fdd2c60157b6bd70ba97968ab81ff34b169127d6441ce3e46" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 164, "digest": "sha256:e18adfafeca523e759b123ac97d8a04bd4e71dbf398d8026e7c501a650277d6c" } ] }
It looks similar to the example Image Manifest we saw at the beginning of this post.
View the amd64 Image Manifest.
marvell@macchiato-1:~$ docker manifest inspect --insecure macchiato-1:5000/my-ubuntu-amd64:17.04 { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 3615, "digest": "sha256:fe1cc5b9183012672af35205799ac5b6a70bc68762011fe82257d5dabf5ba966" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 38640200, "digest": "sha256:c2ca09a1934b951505ecc4d6b2e4ab7f9bf27bcdfb8999d0181deca74daf7683" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 847, "digest": "sha256:d6c3619d2153ffdefa4a9c19f15c5d566ce271b397a84537baa9ee45b24178f2" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 570, "digest": "sha256:0efe07335a049e6afcd757db2d17ba37a12b717eb807acb03ddf3cd756b9fc2a" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 854, "digest": "sha256:6b1bb01b3a3b72463ae8ac5666d57b28f1a21d5256271910ac8df841aa04ecd1" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 163, "digest": "sha256:43a98c1873995475a895f3d79f405232ef5230076b3f610c949c2e8341743af7" } ] }
Again, it looks like a standard Image Manifest.
Now that we know our two images are in the registry, let's create and push the Manifest List. We will name the Manifest List macchiato-1:5000/my-ubuntu:17.04. Notice that we do not add the suffixes of -arm64 or -amd64. This is the point of a Manifest List, they allow machines of different architectures to pull and run the same image name and tag.
macchiato-1:5000/my-ubuntu:17.04
-arm64
-amd64.
marvell@macchiato-1:~$ docker manifest create --insecure macchiato-1:5000/my-ubuntu:17.04 macchiato-1:5000/my-ubuntu-arm64:17.04 macchiato-1:5000/my-ubuntu-amd64:17.04 Created manifest list macchiato-1:5000/my-ubuntu:17.04 marvell@macchiato-1:~$ docker manifest push --insecure macchiato-1:5000/my-ubuntu:17.04 Pushed ref macchiato-1:5000/my-ubuntu@sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459 with digest: sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459 Pushed ref macchiato-1:5000/my-ubuntu@sha256:ea65878d686232eb82eaaffd68676c7334af43446e0005d0c2810f34c9f9d376 with digest: sha256:ea65878d686232eb82eaaffd68676c7334af43446e0005d0c2810f34c9f9d376 sha256:b1db58d235c39e635325aa248c52f0025a9f629aa73606eaeff5939baa51beb1
The --insecure flag is needed for the same reason explained earlier.
Now that the Manifest List is pushed into the registry, let's inspect it.
marvell@macchiato-1:~$ docker manifest inspect --insecure macchiato-1:5000/my-ubuntu:17.04 { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1357, "digest": "sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459", "platform": { "architecture": "amd64", "os": "linux" } }, { "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "size": 1357, "digest": "sha256:ea65878d686232eb82eaaffd68676c7334af43446e0005d0c2810f34c9f9d376", "platform": { "architecture": "arm64", "os": "linux" } } ] }
Under the section labeled manifests, we see the two architectures we want the registry to support for the my-ubuntu:17.04 image.
my-ubuntu:17.04
On the MacchiatoBin, let's try pulling and running macchiato-1:5000/my-ubuntu:17.04. Notice that there's no architecture indicated in the image name and tag.
marvell@macchiato-1:~$ docker run -ti macchiato-1:5000/my-ubuntu:17.04 bash Unable to find image 'macchiato-1:5000/my-ubuntu:17.04' locally 17.04: Pulling from my-ubuntu dec7072e8ec5: Pull complete a52218c98a19: Pull complete 34ce812f23df: Pull complete 9142cb3047db: Pull complete e18adfafeca5: Pull complete Digest: sha256:b1db58d235c39e635325aa248c52f0025a9f629aa73606eaeff5939baa51beb1 Status: Downloaded newer image for macchiato-1:5000/my-ubuntu:17.04 root@bb3e371d4b1b:/#
Try the same on the x86 machine.
ubuntu@x86:~$ docker run -ti macchiato-1:5000/my-ubuntu:17.04 bash Unable to find image 'macchiato-1:5000/my-ubuntu:17.04' locally docker: Error response from daemon: Get https://macchiato-1:5000/v2/: http: server gave HTTP response to HTTPS client.
This error occurs because by default, the Docker daemon on the x86 machine will not communicate with an insecure registry. We will need to configure the Docker daemon to allow for communication with an insecure registry. Notice we did not get this error on the macchiato-1. This is because the registry is running on the macchiato-1.
On the x86 machine write the below to /etc/docker/daemon.json.
/etc/docker/daemon.json.
{ "insecure-registries" : [ "macchiato-1:5000" ] }
Then restart the Docker daemon.
ubuntu@x86:~$ sudo systemctl daemon-reload ubuntu@x86:~$ sudo service docker restart
Try to run the image again.
ubuntu@x86:~$ docker run -ti macchiato-1:5000/my-ubuntu:17.04 bash Unable to find image 'macchiato-1:5000/my-ubuntu:17.04' locally 17.04: Pulling from my-ubuntu c2ca09a1934b: Pull complete d6c3619d2153: Pull complete 0efe07335a04: Pull complete 6b1bb01b3a3b: Pull complete 43a98c187399: Pull complete Digest: sha256:b1db58d235c39e635325aa248c52f0025a9f629aa73606eaeff5939baa51beb1 Status: Downloaded newer image for macchiato-1:5000/my-ubuntu:17.04 root@5398c3f5f7df:/#
At this point, we can be sure that our registry is serving the correct images to AArch64 and x86 machines.
We've discussed Image Manifests, Manifest Lists, and have shown how they are used to setup a multi-arch Docker registry. This is possible because of how mature Docker has become with respect to supporting different HW architectures.
If you are interested in contributing to the Arm ecosystem, we'd like to encourage readers to get involved in the Works on Arm project. This project aims to further develop the Arm Software ecosystem in the data center. The idea of the Works on Arm project is to take Open Source projects, deploy them on Arm platforms, and debug any functional and performance issues found in the process. All it takes to get started is to build and run the SW as intended by its developers. We'd like to encourage readers to get involved with this project.
[CTAToken URL = "https://www.worksonarm.com/" target="_blank" text="Works on Arm Project" class ="green"]