Cross compilation for Arm

There are many ways to compile a software for Arm architecture. If you have a small project and an Arm device with a Linux-based system, you can easily compile it using the shipped compiler on the device. You don't need any complex solution for it. Otherwise, if you have a large source code and want to compile it frequently a different approach is needed. For that case you can use a cross-compiler, which is running on your host system (PC), and the provided binaries are made for your target system (Arm device). The problem is that a complex code is usually relies on different libraries. In that case, you have to provide these headers and libraries for the cross-compiler. Furthermore, the state-of-the-art software has different mechanism to detect the requested libraries, which could make your life difficult if you want to cross compile them. There are many tricks and hacks to make it work, but when a new dependency comes to the project, then you need to find new, occasionally ugly hacks. Unfortunately, there is no elegant and fast solution. In this article, I'd like to show a way, which requires as few hacks as possible.

(Note: I'm going to list several shell commands below. The # means root mode and $ means user mode. Please be careful which mode you are using!)

What is a chroot?

On Unix systems there is a program called chroot. It could change you current root directory to an other, for example to an Arm root directory. Of course you need an emulator as well to run the Arm binaries. That way you can have the "same" environment as on your Arm device and you can exploit the strength of your host machine apart from the emulator overhead.

Qemu and Binfmt

If you try to chroot to an Arm root directory without setting an emulator you may get the following error message:

  # chroot arm-rootdir/
  chroot: failed to run command ‘/bin/bash’: Exec format error

This happens because chroot is trying to execute arm-rootdir/bin/bash by default which is an Arm binary. If you want to run Arm binaries, you will need an emulator. For that purpose Qemu (Quick EMUlator) can be a good choice. It's relatively fast, and it supports many architectures like Arm, MIPS, PowerPC etc. For chrooting, you will need a statically linked Qemu to your host system. If you are using Ubuntu distribution, you can find it in qemu-user-static package or on older systems qemu-kvm-extras-static.

Installing static Qemu on your host:

  # apt-get install qemu-user-static

Now you can verify your qemu-arm-static executable:

  $ file /usr/bin/qemu-arm-static
  /usr/bin/qemu-arm-static: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.15,
  BuildID[sha1]=7fc1a20617692e0601e0faf2594730c8bf718e50, stripped

If you are using other Linux distribution, you can still use the same package. Find and download the package from Ubuntu and get the qemu-arm-static file from it.

  $ ar vx qemu-user-static_1.4.0+dfsg-1expubuntu4_amd64.deb
  $ tar -xzvf data.tar.gz
  $ ls ./usr/bin

Now, you've got an emulator, just one thing missing. Every time you run an Arm binary, your emulator has to be executed with the selected binary. There is a kernel feature called Binfmt, which does exactly this. You can read more about Binfmt.

Mounting binfmt_misc:

  # mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc

Setting the magic string and the emulator that should be invoked with the Arm binaries:

  # echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register

(Note that sudo doesn't work here, because it gives permission for echo not to the > redirection.)

With the help of this command, you set /usr/bin/qemu-arm-static to be invoked every time, when you run an Arm binary. Of course, if you change your root directory to your Arm one, you have to have this emulator available there as well. So, you should have to copy the qemu-arm-static to your Arm root directory.

Get an Arm root directory

There are several ways to get an Arm root directory. You can copy it from your Arm device, or make one with debootsrap or other tools. Here I'm showing you the simplest way I know. Maybe you have heard about Linaro. Among other things, they are publishing Ubuntu images for Arm devices, which are very useful for us. These images contain at least a base Ubuntu system. On Linaro's website you can find the latest Ubuntu images for different devices, and if you need, you can find the older versions as well. These images contain a boot and a root partition. It doesn't really matter which device you choose because you need just the root file system from it and these are almost the same. You can mount the root partition and then you can use it as an Arm root directory. The best thing is about the image, that it will preserve the changes after you umount it.

If you want to run your compiled binaries on your device, you should choose an image with a similar Ubuntu version, what you have on your device.

Download an appealing image:

  $ wget http://releases.linaro.org/13.09/ubuntu/panda/panda-raring_developer_20130922-471.img.gz
  $ gunzip panda-raring_developer_20130922-471.img.gz

You can easily check where the root partition starts:

  $ fdisk -l panda-raring_developer_20130922-471.img

  Disk panda-raring_developer_20130922-471.img: 1073 MB, 1073741824 bytes, 2097152 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disk label type: dos
  Disk identifier: 0x00000000

                                    Device Boot      Start         End      Blocks   Id  System
  panda-raring_developer_20130922-471.img1   *          63      106494       53216    c  W95 FAT32 (LBA)
  panda-raring_developer_20130922-471.img2          106496     2097151      995328   83  Linux

Now you can see the partitions and the size information. For mounting you need the start offset of the second partition (106496) multiplied with the block size (512).

  # mkdir ubuntu-arm/
  # mount -o loop,offset=$[start_offset*block_size] panda-raring_developer_20130922-471.img ubuntu-arm/

Another important thing is to put your emulator into the mounted directory.

  # cp /usr/bin/qemu-arm-static ubuntu-arm/usr/bin/

In ubuntu-arm/ you should have the whole Arm root directory now.

Expand Linaro image

The Linaro images usually not so big (1GB). Sometimes this image size can be too small. In that case there is a way to expand the image. I will show you how.

First, make a 1GB size file with a program called dd:

  $ dd if=/dev/zero of=expand_tmp bs=1M count=1000

If your image is mounted then umount it and add the expand_tmp content (which are just zeros) to the end of your Linaro image:

  $ cat expand_tmp >> linaro.img

Now your image is bigger by 1GB, but the second partition size is the same, thus you need to re-size it.

Mount again the second partition of the image:

  # mount -o loop,offset=$[start_offset*block_size] linaro.img ubuntu-arm/

With the help of the df program, you can check which /dev/loop device you need to resize.

Resizing the partition:

  # resize2fs -f /dev/loop0

The partition is now bigger by 1GB. You can repeat the process whenever you run out of space in your image.

Chroot to Arm root directory

If you want to chroot to your new Arm root directory, you have to prepare it first. You need to mount some important directories from your host system root directory.

Mounting directories:

  # mount --bind /proc ubuntu-arm/proc
  # mount --bind /tmp ubuntu-arm/tmp
  # mount --bind /sys ubuntu-arm/sys
  # mount --bind /dev ubuntu-arm/dev
  # mount --bind /dev/pts ubuntu-arm/dev/pts

You can also mount your host system /home directory and work from there, that way you can keep your Linaro image small and movable.

  # mount --bind /home ubuntu-arm/home

The default user in Linaro images is linaro. You can use that or your host system's user, when you chroot in. Using the linaro user is straightforward so I describe how to use your host user. In the chroot you have to change the linaro username to your username in ubuntu-arm/etc/passwd and copy the line with your username in /etc/shadow to ubuntu-arm/etc/shadow and comment out the linaro one. This way you can use your host system username in the chroot.

It is good to know, whether you're in or outside of the chrooot, in order to make it explicit you can modify your prompt to always display it.

Set a name to the chroot:

  # echo ubuntu-arm > ubuntu-arm/etc/debian_chroot

You may also want to have network connection in your chroot.

Copy your resolv.conf to networking:

  # cp /etc/resolv.conf ubuntu-arm/etc/

After these steps the mounted root directory is ready to use. Let's chroot in it!

  # chroot ubuntu-arm/

If you are lucky and you didn't make any mistake you will get a root shell in your Arm root directory. You may set locales to avoid some warning messages after installing new packages. Set locales:

  (ubuntu-arm) # locale-gen en_US en_US.UTF-8
  (ubuntu-arm) # dpkg-reconfigure locales

Now you can install packages with apt-get and enjoy the brand new Arm environment.

Life after reboot

Most probably you will restart your computer for any reason sometimes. If you want to use your chroot again there are some tasks that you need to do. You can do it by hand or you can make a script for it, it's up to you.

Setup Binfmt again:

  # mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
  # echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register

Mount the Linaro image:

  # mount -o loop,offset=$[start_offset*block_size] linaro.img ubuntu-arm/

Mount the host directories in it:

  # mount --bind /proc ubuntu-arm/proc
  # mount --bind /tmp ubuntu-arm/tmp
  # mount --bind /sys ubuntu-arm/sys
  # mount --bind /dev ubuntu-arm/dev
  # mount --bind /dev/pts ubuntu-arm/dev/pts

Mount your /home in it:

  # mount --bind /home ubuntu-arm/home

After these you can use it again.

Compiling in the Arm environment

Get into the chroot and install the packages what you need.

  # chroot ubuntu-arm/
  (ubuntu-arm) # apt-get install g++

Change from root mode to your host user.

  (ubuntu-arm) # su your_host_username
  (ubuntu-arm) $

Compile and run a HelloWorld program.

  (ubuntu-arm) $ g++ hello.cpp -o hello
  (ubuntu-arm) $ ./hello
  Hello! I'm running on Arm! \o/

Now you can start to compile your favorite desktop project in this Arm environment. If you have any questions, you're welcome to comment or contact me.