Virtualization on ARM with Xen

With ARM entering the server space, a key technology in play in this segment is Virtualization. Virtualization is not a tool solely for servers and the data center, it is also used in the embedded space in segments like automotive and it is also starting to be used in mobile.

This is not a new technology, IBM pioneered it in the 1960s, and there are many different hypervisors implementing different methods of virtualization. In the Open Source realm there are two major hypervisors: KVM and Xen. Both are interact directly with the Linux kernel, however KVM is solely in the Linux domain whereas Xen works with Linux, *BSD and other UNIX variants.

In the past it was generally accepted that there are two types of hypervisor, Type1 (also known as bare metal or native) where the hypervisor runs directly on the host server and controls all aspects of the hardware and manages the guest operating systems, and Type2 (also known as hosted) where the hypervisor runs within a normal operating system; under this classification Xen falls into the Type1 camp and KVM fell into the Type2 camp. However the modern implementations of the hypervisors has now blurred the lines of distinction.

This time round I’ll be taking a look at the Xen Hypervisor, which is now one of the Linux Foundation’s collaborative projects.

Brief overview of some of Xen’s features

• Small footprint. Based on a microkernel design, it has a limited interface to the guest virtual machine and takes up around 1MB of memory.
• Operating system agnostic. Xen works well with BSD variants and other UNIX systems, although most deployments use Linux.
• Driver Isolation. This allows the main device drivers of a system to run inside the VM, which enables the VM containing drivers to be rebooted in the event of a crash or compromise without affecting the host or other guests. In the Xen model the majority of the device drivers run in virtual machines rather than in the hypervisor, as well as allowing reusing of existing OS driver stacks this allows the VM containing the driver to be rebooted without affecting the host of other guests. Individual drivers can even be run in separate VMs in order to improve isolation and fault tolerance or just to take advantage of differing OS functionality.
• Paravirtualization (PV). This style of port enables Xen to run on hardware that doesn’t have virtualization extensions, such as Cortex-A5/A8/A9 in ARM's case. There can also be some performance gains for some PV guests, but this requires the guests to be modified and prevents “out of the box” implementations of operating systems.
• No emulation, no QEMU. Emulated interfaces are slow and insecure. By using hardware virtualization extensions and IO paravirtualization, Xen removes any need for emulation. As a result you have a smaller code base and better performances

The Xen hypervisor runs directly on the hardware and is responsible for handling CPU, Memory, and interrupts. It is the first program running after exiting the bootloader. Virtual machines then run atop of Xen. A running instance of a virtual machine in Xen is called a DomU or guest. The controller for the guest VMs is a special host VM called Dom0 and contains the drivers for all the devices in the system. Dom0 also contains a control stack to manage virtual machine creation, destruction, and configuration.

Pieces of the puzzle

• The Xen Hypervisor is a lean software layer that runs directly on the hardware and as mentioned is responsible for managing CPU, memory, and interrupts. The hypervisor itself has no knowledge of I/O functions such as networking and storage.
• Guest Domains/Virtual Machines (DomU) are virtualized environments, each running their own operating system and applications. On other architectures Xen supports two different virtualization modes: Paravirtualization (PV) and Hardware-assisted or Full Virtualization (HVM). Both guest types can be used at the same time on a single Xen system. On ARM there is only one mode for virtualization that is more of a hybrid of the two, which effectively is hardware based with paravirt extensions. Some call this mode PVHVM. Xen guests are totally isolated from the hardware and have no privilege to access hardware or I/O functionality. Which is where DomU comes in, standing for unprivileged domain.
• The Control Domain (or Dom0) is a specialized Virtual Machine that has special privileges like the capability to access the hardware directly, handles all access to the system’s I/O functions and interacts with the other Virtual Machines. It also exposes a control interface to the outside world, through which the system is controlled. The Xen hypervisor will not function without Dom0, which is the first VM started by the system.
• Toolstack and Console: Dom0 contains a control stack (known as Toolstack) that allows a user to manage virtual machine creation, destruction, and configuration. The toolstack exposes an interface that is either driven by a command line console, by a graphical interface or by a cloud orchestration stack (OpenStack or CloudStack.)
• Xen-enabled operating systems: Running an operating system as Dom0 or DomU requires the operating system kernel to be Xen enabled. However porting an operating system to Xen on ARM is simple: it just needs a few new drivers for the Xen paravirtualized IO interfaces. Existing open source PV drivers in Linux and FreeBSD are likely to be reusable. Linux distributions that are based on a recent Linux kernel (3.8+) are already Xen enabled and usually contain packages for the Xen hypervisor and tools.

The latest version of Xen is 4.4.0, which was released in March and has support for both ARMv7 and ARMv8. For this exercise I’ll be looking at using Xen on ARMv8 with the Foundation Model.

Please consult the Xen Wiki for more information on using Xen with Virtualization Extensions and using Xen with Models. For discussion/review/information and help there are mailinglists.

Development set up

You can use whichever Linux distribution you prefer, so long as you have a suitable cross-compilation environment set up. I’m using openSUSE 13.1 with the Linaro Cross Toolchain for AArch64.

Typographic explanation:

host$ = run as a regular user on host machine

host# = run as root user on host machine (can use sudo if you prefer)

chroot> = run as root user in chroot environment

model> = run as root user in a running Foundation Model

The first steps are to build Xen and a Linux kernel for use in both Dom0 and DomU machines. We then package Xen and Linux along with a Device Tree together for Dom0 to be used in the model using boot-wrapper.

Build Xen

If using Linaro’s toolchain, ensure the /bin directory is in your $PATH

host$ git clone git://xenbits.xen.org/xen.git xen

host$ cd xen

host$ git checkout RELEASE-4.4.0

There is a small build bug due to use of older autotools which will be fixed in the 4.4.1 release. Rather than wait for the next release, we’ll just backport it now.

host$ git cherry-pick 0c68ddf3085b90d72b7d3b6affd1fe8fa16eb6be

There is also a small bug in GCC with PSR_MODE see bug LP# 1169164. Download the attached PSR_MODE_workaround.patch

host$ patch -i PSR_MODE_workaround.patch -p1

host$ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- CONFIG_EARLY_PRINT=fastmodel

host$ cd ..

Build Linux

host$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

host$ cd linux

host$ git checkout v3.13

Create a new kernel config:

host$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig

host$ sed -e 's/.*CONFIG_XEN is not set/CONFIG_XEN=y/g' -i .config

host$ sed -e 's/.*CONFIG_BLK_DEV_LOOP is not set/CONFIG_BLK_DEV_LOOP=y/g' -i .config

host$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- oldconfig

Make sure to select Y to all Xen config options

I have attached a kernel.config which has all the required options enabled for reference.

host$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image

host$ cd ..

Obtain the Foundation Model

In a browser go to Arm Developer's Fixed Virtual Platforms page.

Scroll to the bottom and select “Download Now”

This should provide FM000-KT-00035-r0p8-52rel06.tgz

Extract the tarball

host$ tar xaf FM000-KT-00035-r0p8-52rel06.tgz

Build Boot Wrapper and device tree

It is common to run the models without real firmware. In this case a boot-wrapper is needed to provide a suitable boot time environment for Xen, which allows booting into Non-Secure HYP mode and providing boot modules etc.:

host$ git clone -b xen-arm64 git://xenbits.xen.org/people/ianc/boot-wrapper-aarch64.git

host$ cd boot-wrapper-aarch64

host$ ln -s ../xen/xen/xen Xen

host$ ln -s ../linux/arch/arm64/boot/Image Image

Use the attached foundation-v8.dts to build the device tree blob

host$ dtc -O dtb -o fdt.dtb foundation-v8.dts

host$ make CROSS_COMPILE=aarch64-linux-gnu- FDT_SRC=foundation-v8.dts IMAGE=xen-system.axf

host$ cd ..

Run the Model to make sure the kernel functions, it will panic as we haven’t setup the rootfs yet:

host$ ./Foundation_v8pkg/models/Linux64_GCC-4.1/Foundation_v8 \

     --image boot-wrapper-aarch64/xen-system.axf

Chroot Build Environment

Next we create a suitable chroot build environment using the AArch64 port of openSUSE. We will use the qemu-user-static support for AArch64 to run the chroot on the (x86) host.

First we build the qemu binary, then construct the chroot, finally we build Xen in the chroot environment.

Building qemu-aarch64-user

host$ git clone https://github.com/openSUSE/qemu.git qemu-aarch64

host$ cd qemu-aarch64

host$ git checkout aarch64-work

Install some build dependencies:

host# zypper in glib2-devel-static glibc-devel-static libattr-devel-static libpixman-1-0-devel ncurses-devel pcre-devel-static zlib-devel-static

host$ ./configure --enable-linux-user --target-list=arm64-linux-user --disable-werror --static

host$ make -j4

host$ ldd ./arm64-linux-user/qemu-arm64

     not a dynamic executable

This last step is to verify that the resulting binary is indeed a static binary. We will copy it into the chroot later on.

We now need to enlighten binfmt misc about aarch64 binaries:

On openSUSE:

host# cp scripts/qemu-binfmt-conf.sh /usr/sbin/

host# chmod +x /usr/sbin/qemu-binfmt-conf.sh

host# qemu-binfmt-conf.sh

On Debian:

host# update-binfmts --install aarch64 /usr/bin/qemu-aarch64-static \

   --magic '\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7' \

   --mask '\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff'

host$ cd ..

Build the chroot environment

host$ wget http://download.opensuse.org/ports/aarch64/distribution/13.1/appliances/openSUSE-13.1-ARM-JeOS.aarch64-rootfs.aarch64-1.12.1-Build32.2.tbz

Note: the file name may change due to the continuous image building, if the above does not work verify the latest version of the tarball here.

host$ mkdir aarch64-chroot

host# tar -C aarch64-chroot -xaf openSUSE-12.3-AArch64-JeOS-rootfs.aarch64-1.12.1-Build5.13.tbz

Install the qemu binary into the chroot environment

host# cp qemu-aarch64/arm64-linux-user/qemu-arm64 aarch64-chroot/usr/bin/qemu-aarch64-static

host# cp /etc/resolv.conf aarch64-chroot/etc/resolv.conf

Build the Xen tools in the chroot environment (finally)

Copy the Xen sources into the chroot

host# cp -r xen aarch64-chroot/root/xen

Chroot into the aarch64 environment

host# chroot aarch64-chroot /bin/sh

We now need to install some build dependencies

chroot> zypper install gcc make patterns-openSUSE-devel_basis git vim libyajl-devel python-devel wget libfdt1-devel libopenssl-devel

If prompted to trust the key, I’ll let you choose whether to trust permanently or just this time (personally I chose to always trust the key).

chroot> cd /root/xen

chroot> ./configure

chroot> make dist-tools

chroot> exit

The Xen tools are now in aarch64-chroot/root/xen/dist/install

Root filesystem and image

We will create an ext3 formatted filesystem image, we will also use a simplified initscript to avoid long waits while running the model.

host$ wget http://download.opensuse.org/

This is the same rootfs tarball as use for the chroot. You can re-use the previously downloaded tarball if you wish.

host$ dd if=/dev/zero bs=1M count=1024 of=rootfs.img

host$ /sbin/mkfs.ext3 rootfs.img

Say yes, we know it’s not a block device

host# mount -o loop rootfs.img /mnt

host# tar -C /mnt -xaf openSUSE-13.1-ARM-JeOS.aarch64-rootfs.aarch64-1.12.1-Build32.2.tbz

Install the Xen tools that we built earlier

host# rsync -aH aarch64-chroot/root/xen/dist/install/ /mnt/

Create the init script

host# cat > /mnt/root/init.sh <<EOF

#!/bin/sh

set -x

mount -o remount,rw /

mount -t proc none /proc

mount -t sysfs none /sys

mount -t tmpfs none /run

mkdir /run/lock

mount -t devtmpfs dev /dev

/sbin/udevd --daemon

udevadm trigger --action=add

mkdir /dev/pts

mount -t devpts none /dev/pts

mknod -m 640 /dev/xconsole p

chown root:adm /dev/xconsole

/sbin/klogd -c 1 -x

/usr/sbin/syslogd

cd /root

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

exec /bin/bash

EOF

host# chmod +x /mnt/root/init.sh

Get missing runtime dependencies for Xen

host$ wget http://download.opensuse.org/ports/aarch64/distribution/13.1/repo/oss/suse/aarch64/libyajl2-2.0.1-14.1.2.aarch64.rpm

host$ wget http://download.opensuse.org/ports/aarch64/distribution/13.1/repo/oss/suse/aarch64/libfdt1-1.4.0-2.1.3.aarch64.rpm

host# cp libyajl2-2.0.1-14.1.2.aarch64.rpm libfdt1-1.4.0-2.1.3.aarch64.rpm /mnt/root/

host# umount /mnt

Start the model

host$ ./Foundation_v8pkg/models/Linux64_GCC-4.1/Foundation_v8 \

     --image boot-wrapper-aarch64/xen-system.axf \

     --block-device rootfs.img \

     --network=nat

Silence some of the harmless warnings

model> mkdir /lib/modules/$(uname -r)

model> depmod -a

Install the runtime dependencies:

model> rpm -ivh libfdt1-1.3.0-9.1.1.aarch64.rpm libyajl2-2.0.1-12.1.1.aarch64.rpm

model> ldconfig

Start the Xen daemon, you can ignore the harmless message about i386 qemu if it appears.

model> /etc/init.d/xencommons start

If you get an error of missing file for /etc/init.d/xencommons re-run ldconfig.

Confirm that Dom0 is up:

model> xl list

Name                                        ID Mem VCPUs      State   Time(s)

Domain-0 0   512     2 r-----      13.9

Congratulations, you now have a working Xen toolstack. You can now shut down the model for now.

Creation of a DomU guest

For the guest rootfs we will use a smaller OpenEmbedded based Linaro image rather than a full openSUSE image purely for space constraints.

host$ wget http://releases.linaro.org/latest/openembedded/aarch64/linaro-image-minimal-genericarmv8-20140223-649.rootfs.tar.gz

host$ dd if=/dev/zero bs=1M count=128 of=domU.img

host$ /sbin/mkfs.ext3 domU.img

Again say yes, we know it’s not a block device

host# mount -o loop domU.img /mnt

host# tar -C /mnt -xaf linaro-image-minimal-genericarmv8-20140223-649.rootfs.tar.gz

host# umount /mnt

Make the DomU rootfs and kernel available to the Dom0

host# mount -o loop rootfs.img /mnt

host# cp domU.img /mnt/root/domU.img

host# cp linux/arch/arm64/boot/Image /mnt/root/Image

Create the config for the guest

host# cat > /mnt/root/domU.cfg <<EOF

kernel = "/root/Image"

name = "guest"

memory = 512

vcpus = 1

extra = "console=hvc0 root=/dev/xvda ro"

disk = [ 'phy:/dev/loop0,xvda,w' ]

EOF

host# umount /mnt

Start the model again:

host$ ./Foundation_v8pkg/models/Linux64_GCC-4.1/Foundation_v8 \

     --image boot-wrapper-aarch64/xen-system.axf \

     --block-device rootfs.img \

            --network=nat

model> losetup /dev/loop0 domU.img

model> /etc/init.d/xencommons start

Create the DomU using the config

model> xl create domU.cfg

View the guest’s info on the Xen console

model> xl list

Screenshot of the Dom0 host

Start the Dom0

model> xl console guest

Screenshot of the DomU guest

Now all that’s left is to have a lot of fun!