Spawn a Linux virtual machine on Arm using QEMU (KVM)

Preface

This article outlines how to utilize virtualization on Armv8 systems using QEMU. These instructions were primarily written for the Armv8 Base Platform FVP but are easily adapted to other platforms such as the Juno development board.

Prerequisites

Follow the instructions here to initialize a new built from source OpenEmbedded workspace containing all of the required sources and tools.  This directory will be referred to as <workspace> from this point onward.

+-------------+--------------------------------------------------+
| Workspace   | <workspace>                                      |
| Platform    | Armv8-A Base Platform with 64-bit software stack |
| Type        | Build from source                                |
| Environment | Linux kernel + filesystem                        |
| Kernel      | ack-4.9-armlt (Android Common Kernel)            |
| Filesystem  | Open Embedded Minimal 15.09                      |
+-------------+--------------------------------------------------+

Install the Armv8 Base Platform model from Arm Developer.

Install the libncurses5 and kpartx packages.

Configure the kernel Image to support virtualization by editing the kernel config flags. To do so follow the instructions here. Specifically set CONFIG_VIRTUALIZATION=y and then apply the following changes:

Installing QEMU

In order to run QEMU on the Armv8 FVP a static version of QEMU needs to be created. To do that the run the following command sequence

### Clone the QEMU Github repository onto the workspae directory
$ cd <workspace>
$ git clone git://git.qemu-project.org/qemu.git

Navigate to the newly created qemu directory and initialize and update all the modules
$ cd qemu
$ git submodule init
$ git submodule update --recursive
$ git submodule status --recursive
$ git checkout stable-2.9

### Create a folder to contain the build and navigate into it
$ mkdir build
$ cd build

### Configure QEMU and run 'make' to complete the build
$ ../configure --target-list=aarch64-softmmu --with-coroutine=gthread --enable-fdt --enable-kvm --disable-werror --audio-drv-list="" --static 
$ make

This will create a static binary of QEMU located in <workspace>/qemu/build/aarch64-softmmu/

WARNING: If during the build process the error "error: static declaration of ‘memfd_create’ follows non-static declaration static int memfd_create(const char *name, unsigned int flags)" occurs then the script <workspace>/qemu/utils/memfd.c needs to be edited so that the variable declaration 'static int memfd_create' is renamed to 'static int tmp_memfd_create' 

At some case you may have to cross-compile the static binary once again for arm64 or build qemu on arm hardware (in contrast to something like a VM with an Intel processor). Alternatively you can pass the --cpu=aarch64 option when configuring QEMU.

Configuring the host and guest filesystems

To create the host filesystems we need to:

  • 1. Create a filesystem image for the host
  • 2. Mount the host filesystem using NFS and copy the necessary components to launch QEMU on the host's filesystem root directory
  • 3. Unmount the host filesystem

You can optionally repeat this for the guest, though we recommend using a prebuilt BusyBox image obtained by following the instructions here.

First create the host filesystem. This can be done with Buildroot as described in Part I of this tutorial or alternatively, a minimal filesystem for Armv8 can be downloaded here. Then use the instructions on Part II of the same tutorial to create an empty filesystem image and extract the filesystem in it.

Tip: For the guest it is advisable to use a prebuilt configuration of Busybox by following the instructions here.

### Mount the host filesystem image
$ sudo mkdir /mnt/mydrive
$ sudo kpartx -a /path/to/hostfs.img 
$ sudo mount /dev/mapper/loopXp1 /mnt/mydrive

### Copy the qemu binary into the root directory
$ sudo cp <workspace>/qemu/build/aarch64-softmmu/qemu-system-aarch64 /mnt/mydrive/home/root

### If you are using an Arm Platforms Deliverables configuration build for the guest, copy the necessary files
### from the platform specific directory onto the root directory of the host. For Busybox:
$ sudo cp /path/to/fvp-latest-busybox-uboot/Image /mnt/mydrive/home/root
$ sudo cp /path/to/fvp-latest-busybox-uboot/ramdisk.img /mnt/mydrive/home/root

### (OPTIONAL) If you have created a custom filesystem Image for the guest then copy that as well 
$ sudo cp /path/to/guestfs.img /mnt/mydrive/home/root

### Unmount the host filesystem image
$ sudo umount /mnt/mydrive && rmdir /mnt/mydrive
$ sudo kpartx -d /path/to/hostfs.img

Launching Guest using QEMU

After you have launched the Armv8 FVP as described here and are in the root directory you should be able to see all the necessary files needed to launch the guest, namely:

  • ramdisk.img
  • Image
  • guestfs.img
  • qemu-system-aarch64 binary

To launch the guest simply run the static QEMU binary using the following command

### To run a custom guest configuration with a filesystem 'guestfs.img run the following:
$ sudo ./qemu-system-aarch64 -enable-kvm -nographic \
-kernel Image -initrd ramdisk.img \
-drive if=none,file=guestfs.img,id=fs \
-device virtio-blk-device,drive=fs -m 512 -M virt \
-cpu host -append "root=/dev/vda1 rw rootwait"

### To run a Busybox guest run the following: 
$ sudo ./qemu-system-aarch64 -enable-kvm -nographic \
-kernel Image -initrd ramdisk.img -m 512 -M virt -cpu host

Instruction Delta for Juno

This section highlights the differences between the above instructions, which target the Armv8 FVP's, and the steps necessary to achieve virtualization on the Juno hardware development platform. Although the exact same instructions as above can be replicated on the Juno, an easier way is outlined below.

Installing QEMU

QEMU can be installed as a package from the Ubuntu repository.

   $ sudo apt-get install qemu-system-aarch64

This will download an executable for QEMU along with all the dependencies and required packages

Configuring the guest filesystem

Since the virtualization will be ran natively on the Juno there is no need to create a host filesystem as done with the FVP since the Juno filesystem will be used as the host filesystem. As such it is necessary to just create a guest configuration of your choice. This includes the Kernel Image, ramdisk image and the filesystem image (if applicable).

Running the guest

To run the guest run the the 'qemu-system-aarch64' command with the following arguments

### To run a custom guest configuration with a filesystem 'guestfs.img run the following:
$ sudo qemu-system-aarch64 -enable-kvm -nographic \
-kernel /path/to/Image \
-drive if=none,file=/path/to/guestfs.img,id=fs \
-device virtio-blk-device,drive=fs -m 512 -M virt \
-cpu host -append "root=/dev/vda1 rw rootwait"

### To run a configuration from Arm Platforms Deliverable point to the platform specific output
### For a Busybox guest run the following: 
$ sudo qemu-system-aarch64 -enable-kvm -nographic -kernel /path/to/fvp-latest-busybox-uboot/Image \
    -initrd /path/to/fvp-latest-busybox-uboot/ramdisk.img -m 512 -M virt -cpu host
Related
Recommended