Deploying a Multi-Arch Docker Registry

In this post, we'll discuss Image Manifests, Manifest Lists, and their role in enabling multi-architecture Docker registries. Then we'll walk through an example of setting up a multi-architecture Docker registry on a MacchiatoBin. Although we are using the MacchiatoBin in this blog, the instructions presented below will work on just about any AArch64 HW available on the market. The setup of a MacchiatoBin was discussed in Configuring The MacchiatoBin For Kubernetes and Swarm.

Image Manifests and Manifest Lists

An Image Manifest is a file that contains information about the composition of a particular image. Details on Image Manifests can be found on the Image Manifest V2 description page. An example Image Manifest is shown below.

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 7023,
        "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 32654,
            "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f"
        },
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 16724,
            "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
        },
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 73109,
            "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
        }
    ]
}

The key portion of this file is the section labeled layers. It lists all the layers that make up the image. The first layer is the base image, and the subsequent layers are the additional layers needed to compose the final image. This file is automatically created in the registry when an image is pushed into it.

A Manifest List is a file that contains a list of Image Manifests. Below is the Manifest List for the Docker Hub image ubuntu:16.04.

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1357,
         "digest": "sha256:9b47044b1e79b965a8e1653e7f9c04b5f63e00b9161bedd5baef69bb8b4c4834",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1357,
         "digest": "sha256:6741f93913a11e1be518f95acd00ef35b3408e12a9a39db4ffe757a9cec94f1b",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1357,
         "digest": "sha256:8aecae775e1f81d3889929ef15647187b414c833b0798d060bfd778bee668ced",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1357,
         "digest": "sha256:14aa215f96eb6996a323f4290aea3b935dd69636ef8f04e82812ac2bbe277bb9",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1357,
         "digest": "sha256:55f36465901b2f01fcdceecb69d0bc6aacd1c344d01ae25fba6e001f9f7b2dfb",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

The key portion of this file is the section labeled manifests. This section lists six Image Manifests. The main difference between the listed Image Manifests is the architecture. On closer inspection, we can see that two of these Image Manifests are for Arm Ubuntu images.

The Manifest List is how Docker Hub supports different architectures with the single image name and tag of ubuntu:16.04. When a client requests to pull ubuntu:16.04, the registry will check the Manifest List to see if there is an image that matches the client's architecture. If there is, the registry will serve that version of the image to the client.

Setting Up A Multi-Architecture Docker Registry

Let's walk through setting up a registry on the MacchiatoBin, then pull images from the registry from both the MacchiatoBin and an x86 machine. This will verify that the registry can serve multi-architecture images.

Installing Docker and Enabling Experimental Mode

Install the latest stable version of Docker. This can be done by running the following command on both the MacchiatoBin and on the x86 machine.

curl -sSL https://get.docker.com | sh

Check the version that was installed.

marvell@macchiato-1:~$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:32:42 2018
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:30:47 2018
  OS/Arch:          linux/arm64
  Experimental:     false

Notice that both the client and server have their experimental flags set to false. We will need the client's (Docker CLI) experimental flag set to true to use the docker manifest commands. The docker manifest commands are used to create and push the Manifest List. We will also need the server's (Docker Daemon) experimental flag set to true. This will allow the use of the --platform flag with the docker pull command (explained later).

Before we turn on the experimental features, let's try the docker manifest and docker pull commands to verify that we need the experimental features enabled.

marvell@macchiato-1:~$ docker manifest
docker manifest is only supported on a Docker cli with experimental cli features enabled

marvell@macchiato-1:~$ docker pull --platform arm64 ubuntu:17.04
"--platform" is only supported on a Docker daemon with experimental features enabled

As expected, we need to enable experimental features.

To turn on the client experimental features. Add the below to the file ~/.docker/config.json.

{
    "experimental": "enabled"
}

To turn on the daemon experimental features. Add the below to the file /etc/docker/daemon.json.

{
    "experimental": true
}

Notice that the value for the experimental key is different in each case.

For the changes to take effect on the Docker daemon, we need to restart it.

sudo systemctl daemon-reload
sudo service docker restart

If we check the Docker version again, we'll see that the experimental features have been enabled.

marvell@macchiato-1:~$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:32:42 2018
 OS/Arch:           linux/arm64
 Experimental:      true

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:30:47 2018
  OS/Arch:          linux/arm64
  Experimental:     true

Build and Run The Registry

The current Docker registry repo builds for x86 only. This is because the repo contains an x86 pre-built binary that gets copied into the image. As I've mentioned in previous blogs (Architecture Agnostic Container Build Systems and Cloud Management Tools on Arm), this is a practice that should be avoided. Fortunately, there is a PR which addresses this issue. We will use this PR to build a Docker registry image for AArch64.

Start by cloning the registry image repo.

git clone https://github.com/docker/distribution-library-image.git
cd distribution-library-image

Next, we'll fetch the PR, and checkout the branch that is created after the fetch.

git fetch origin pull/74/head:multiarch
git checkout multiarch

Now run the update.sh script. We probably don't have to run it, but it won't hurt. The script just ensures that you will build the latest registry image. I think this script might just be a utility for the project maintainers. 

marvell@macchiato-1:~/distribution-library-image$ ./update.sh 2.6
Updating distribution 2.6...
Full version: 2.6.2
Done.

As shown above, the version we will build is 2.6.2 which is the latest release of the registry.

Run the docker build command to build the registry.

marvell@macchiato-1:~/distribution-library-image$ docker build -t registry:2.6.2 ./2.6/
marvell@macchiato-1:~/distribution-library-image$ docker images | grep registry
registry                                 2.6.2               7602ea1af04f        31 seconds ago      30.7MB

Finally, to run the registry, we'll follow the deploy a registry server instructions. To keep things simple, we will run the registry in an insecure manner. That is, we will not use TLS to secure communication with the registry.

marvell@macchiato-1:~/distribution-library-image$  docker run -d -p 5000:5000 --restart=always --name registry registry:2.6.2
marvell@macchiato-1:~/distribution-library-image$ cd ~
marvell@macchiato-1:~$

You can run docker ps to verify that the container is running.

Push Architecture Specific Images To The Registry

Now that the registry is running, we'll need to push AArch64 and x86 images into the registry. Rather than creating a Dockerfile to build the images, we'll take a shortcut. We'll download the AArch64 and x86 Ubuntu images from Docker Hub, retag them, and then push them into our private registry.

Download the AArch64 image, retag it, and push it to the registry.

marvell@macchiato-1:~$ docker pull ubuntu:17.04
marvell@macchiato-1:~$ docker tag ubuntu:17.04 macchiato-1:5000/my-ubuntu-arm64:17.04
marvell@macchiato-1:~$ docker rmi ubuntu:17.04
marvell@macchiato-1:~$ docker push macchiato-1:5000/my-ubuntu-arm64:17.04
...
17.04: digest: sha256:ea65878d686232eb82eaaffd68676c7334af43446e0005d0c2810f34c9f9d376 size: 1357

Download the x86 image, retag it, and push it to the registry. Notice that we use the --platform flag. This informs the registry (Docker Hub) that we want an x86 (amd64) image even though our client is AArch64.

marvell@macchiato-1:~$ docker pull --platform amd64 ubuntu:17.04
marvell@macchiato-1:~$ docker tag ubuntu:17.04 macchiato-1:5000/my-ubuntu-amd64:17.04
marvell@macchiato-1:~$ docker rmi ubuntu:17.04
marvell@macchiato-1:~$ docker push macchiato-1:5000/my-ubuntu-amd64:17.04
marvell@macchiato-1:~$ docker rmi macchiato-1:5000/my-ubuntu-amd64:17.04
...
17.04: digest: sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459 size: 1357

When we tagged the images, noticed that we prefixed the image name with the machine name that is hosting our registry. This will tell Docker that the image belongs to a private registry. In this case, it's a private registry hosted on port 5000 on macchiato-1. Also, recall that when the image is pushed to the registry, an Image Manifest is automatically created on the registry. We'll explore this a little later.

Last, remove the local copies of the images that were just pushed. This ensures we are only interacting with the images that are hosted on our registry.

marvell@macchiato-1:~$ docker rmi macchiato-1:5000/my-ubuntu-arm64:17.04
marvell@macchiato-1:~$ docker rmi macchiato-1:5000/my-ubuntu-amd64:17.04
marvell@macchiato-1:~$ docker system prune

Create A Manifest List And Push It To The Registry

Before we create the Manifest List, let's take a look at the Image Manifests for the two images we just pushed. To do this, we'll use the docker manifest inspect command. We will also need to add the --insecure flag to the command. This is because the docker manifest inspect command only communicates with the Docker registry. It does not query the client's Docker daemon to figure out which insecure registries are allowed.

View the arm64 Image Manifest.

marvell@macchiato-1:~$ docker manifest inspect --insecure macchiato-1:5000/my-ubuntu-arm64:17.04
{
        "schemaVersion": 2,
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "size": 3614,
                "digest": "sha256:96ad946778d8c777aeb78278e22fb0c1c6de37315e6bfce21654210dc648d789"
        },
        "layers": [
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 35796134,
                        "digest": "sha256:dec7072e8ec5718e527d5c9de1313867b72446297190172e1e3d9de0c5b830ee"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 854,
                        "digest": "sha256:a52218c98a190f0cb50bd8f73946e55694027b173c9a6efcb59cfb76b4e3668c"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 539,
                        "digest": "sha256:34ce812f23df9ecae2602041e53b5b15980e5521f521ac7a4833f848854d03ff"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 857,
                        "digest": "sha256:9142cb3047dbb70fdd2c60157b6bd70ba97968ab81ff34b169127d6441ce3e46"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 164,
                        "digest": "sha256:e18adfafeca523e759b123ac97d8a04bd4e71dbf398d8026e7c501a650277d6c"
                }
        ]
}

It looks similar to the example Image Manifest we saw at the beginning of this post.

View the amd64 Image Manifest.

marvell@macchiato-1:~$ docker manifest inspect --insecure macchiato-1:5000/my-ubuntu-amd64:17.04
{
        "schemaVersion": 2,
        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
        "config": {
                "mediaType": "application/vnd.docker.container.image.v1+json",
                "size": 3615,
                "digest": "sha256:fe1cc5b9183012672af35205799ac5b6a70bc68762011fe82257d5dabf5ba966"
        },
        "layers": [
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 38640200,
                        "digest": "sha256:c2ca09a1934b951505ecc4d6b2e4ab7f9bf27bcdfb8999d0181deca74daf7683"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 847,
                        "digest": "sha256:d6c3619d2153ffdefa4a9c19f15c5d566ce271b397a84537baa9ee45b24178f2"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 570,
                        "digest": "sha256:0efe07335a049e6afcd757db2d17ba37a12b717eb807acb03ddf3cd756b9fc2a"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 854,
                        "digest": "sha256:6b1bb01b3a3b72463ae8ac5666d57b28f1a21d5256271910ac8df841aa04ecd1"
                },
                {
                        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
                        "size": 163,
                        "digest": "sha256:43a98c1873995475a895f3d79f405232ef5230076b3f610c949c2e8341743af7"
                }
        ]
}

Again, it looks like a standard Image Manifest.

Now that we know our two images are in the registry, let's create and push the Manifest List. We will name the Manifest List macchiato-1:5000/my-ubuntu:17.04. Notice that we do not add the suffixes of -arm64 or -amd64. This is the point of a Manifest List, they allow machines of different architectures to pull and run the same image name and tag.

marvell@macchiato-1:~$ docker manifest create --insecure macchiato-1:5000/my-ubuntu:17.04 macchiato-1:5000/my-ubuntu-arm64:17.04 macchiato-1:5000/my-ubuntu-amd64:17.04
Created manifest list macchiato-1:5000/my-ubuntu:17.04
marvell@macchiato-1:~$ docker manifest push --insecure macchiato-1:5000/my-ubuntu:17.04
Pushed ref macchiato-1:5000/my-ubuntu@sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459 with digest: sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459
Pushed ref macchiato-1:5000/my-ubuntu@sha256:ea65878d686232eb82eaaffd68676c7334af43446e0005d0c2810f34c9f9d376 with digest: sha256:ea65878d686232eb82eaaffd68676c7334af43446e0005d0c2810f34c9f9d376
sha256:b1db58d235c39e635325aa248c52f0025a9f629aa73606eaeff5939baa51beb1

The --insecure flag is needed for the same reason explained earlier.

Now that the Manifest List is pushed into the registry, let's inspect it.

marvell@macchiato-1:~$ docker manifest inspect --insecure macchiato-1:5000/my-ubuntu:17.04
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1357,
         "digest": "sha256:213e05583a7cb8756a3f998e6dd65204ddb6b4c128e2175dcdf174cdf1877459",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1357,
         "digest": "sha256:ea65878d686232eb82eaaffd68676c7334af43446e0005d0c2810f34c9f9d376",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

Under the section labeled manifests, we see the two architectures we want the registry to support for the my-ubuntu:17.04 image.

Test The Multi-Architecture Registry

On the MacchiatoBin, let's try pulling and running macchiato-1:5000/my-ubuntu:17.04. Notice that there's no architecture indicated in the image name and tag.

marvell@macchiato-1:~$ docker run -ti macchiato-1:5000/my-ubuntu:17.04 bash
Unable to find image 'macchiato-1:5000/my-ubuntu:17.04' locally
17.04: Pulling from my-ubuntu
dec7072e8ec5: Pull complete
a52218c98a19: Pull complete
34ce812f23df: Pull complete
9142cb3047db: Pull complete
e18adfafeca5: Pull complete
Digest: sha256:b1db58d235c39e635325aa248c52f0025a9f629aa73606eaeff5939baa51beb1
Status: Downloaded newer image for macchiato-1:5000/my-ubuntu:17.04
root@bb3e371d4b1b:/#

Try the same on the x86 machine.

ubuntu@x86:~$ docker run -ti macchiato-1:5000/my-ubuntu:17.04 bash
Unable to find image 'macchiato-1:5000/my-ubuntu:17.04' locally
docker: Error response from daemon: Get https://macchiato-1:5000/v2/: http: server gave HTTP response to HTTPS client.

This error occurs because by default, the Docker daemon on the x86 machine will not communicate with an insecure registry. We will need to configure the Docker daemon to allow for communication with an insecure registry. Notice we did not get this error on the macchiato-1. This is because the registry is running on the macchiato-1.

On the x86 machine write the below to /etc/docker/daemon.json.

{
    "insecure-registries" : [ "macchiato-1:5000" ]
}

Then restart the Docker daemon.

ubuntu@x86:~$ sudo systemctl daemon-reload
ubuntu@x86:~$ sudo service docker restart

Try to run the image again.

ubuntu@x86:~$ docker run -ti macchiato-1:5000/my-ubuntu:17.04 bash
Unable to find image 'macchiato-1:5000/my-ubuntu:17.04' locally
17.04: Pulling from my-ubuntu
c2ca09a1934b: Pull complete
d6c3619d2153: Pull complete
0efe07335a04: Pull complete
6b1bb01b3a3b: Pull complete
43a98c187399: Pull complete
Digest: sha256:b1db58d235c39e635325aa248c52f0025a9f629aa73606eaeff5939baa51beb1
Status: Downloaded newer image for macchiato-1:5000/my-ubuntu:17.04
root@5398c3f5f7df:/#

At this point, we can be sure that our registry is serving the correct images to AArch64 and x86 machines.

Closing Remarks

We've discussed Image Manifests, Manifest Lists, and have shown how they are used to setup a multi-arch Docker registry. This is possible because of how mature Docker has become with respect to supporting different HW architectures.

If you are interested in contributing to the Arm ecosystem, we'd like to encourage readers to get involved in the Works on Arm project. This project aims to further develop the Arm Software ecosystem in the data center. The idea of the Works on Arm project is to take Open Source projects, deploy them on Arm platforms, and debug any functional and performance issues found in the process. All it takes to get started is to build and run the SW as intended by its developers. We'd like to encourage readers to get involved with this project.

Works on Arm Project

Anonymous