There are many ways to compile a software for Arm architecture. If you have a small project and an Arm device with a Linux-based system, you can easily compile it using the shipped compiler on the device. You don't need any complex solution for it. Otherwise, if you have a large source code and want to compile it frequently a different approach is needed. For that case you can use a cross-compiler, which is running on your host system (PC), and the provided binaries are made for your target system (Arm device). The problem is that a complex code is usually relies on different libraries. In that case, you have to provide these headers and libraries for the cross-compiler. Furthermore, the state-of-the-art software has different mechanism to detect the requested libraries, which could make your life difficult if you want to cross compile them. There are many tricks and hacks to make it work, but when a new dependency comes to the project, then you need to find new, occasionally ugly hacks. Unfortunately, there is no elegant and fast solution. In this article, I'd like to show a way, which requires as few hacks as possible.
(Note: I'm going to list several shell commands below. The # means root mode and $ means user mode. Please be careful which mode you are using!)
On Unix systems there is a program called chroot. It could change you current root directory to an other, for example to an Arm root directory. Of course you need an emulator as well to run the Arm binaries. That way you can have the "same" environment as on your Arm device and you can exploit the strength of your host machine apart from the emulator overhead.
If you try to chroot to an Arm root directory without setting an emulator you may get the following error message:
# chroot arm-rootdir/ chroot: failed to run command ‘/bin/bash’: Exec format error
This happens because chroot is trying to execute arm-rootdir/bin/bash by default which is an Arm binary. If you want to run Arm binaries, you will need an emulator. For that purpose Qemu (Quick EMUlator) can be a good choice. It's relatively fast, and it supports many architectures like Arm, MIPS, PowerPC etc. For chrooting, you will need a statically linked Qemu to your host system. If you are using Ubuntu distribution, you can find it in qemu-user-static package or on older systems qemu-kvm-extras-static.
Installing static Qemu on your host:
# apt-get install qemu-user-static
Now you can verify your qemu-arm-static executable:
$ file /usr/bin/qemu-arm-static /usr/bin/qemu-arm-static: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.6.15, BuildID[sha1]=7fc1a20617692e0601e0faf2594730c8bf718e50, stripped
If you are using other Linux distribution, you can still use the same package. Find and download the package from Ubuntu and get the qemu-arm-static file from it.
$ ar vx qemu-user-static_1.4.0+dfsg-1expubuntu4_amd64.deb $ tar -xzvf data.tar.gz $ ls ./usr/bin
Now, you've got an emulator, just one thing missing. Every time you run an Arm binary, your emulator has to be executed with the selected binary. There is a kernel feature called Binfmt, which does exactly this. You can read more about Binfmt.
Mounting binfmt_misc:
# mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
Setting the magic string and the emulator that should be invoked with the Arm binaries:
# echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register
(Note that sudo doesn't work here, because it gives permission for echo not to the > redirection.)
With the help of this command, you set /usr/bin/qemu-arm-static to be invoked every time, when you run an Arm binary. Of course, if you change your root directory to your Arm one, you have to have this emulator available there as well. So, you should have to copy the qemu-arm-static to your Arm root directory.
There are several ways to get an Arm root directory. You can copy it from your Arm device, or make one with debootsrap or other tools. Here I'm showing you the simplest way I know. Maybe you have heard about Linaro. Among other things, they are publishing Ubuntu images for Arm devices, which are very useful for us. These images contain at least a base Ubuntu system. On Linaro's website you can find the latest Ubuntu images for different devices, and if you need, you can find the older versions as well. These images contain a boot and a root partition. It doesn't really matter which device you choose because you need just the root file system from it and these are almost the same. You can mount the root partition and then you can use it as an Arm root directory. The best thing is about the image, that it will preserve the changes after you umount it.
If you want to run your compiled binaries on your device, you should choose an image with a similar Ubuntu version, what you have on your device.
Download an appealing image:
$ wget http://releases.linaro.org/13.09/ubuntu/panda/panda-raring_developer_20130922-471.img.gz $ gunzip panda-raring_developer_20130922-471.img.gz
You can easily check where the root partition starts:
$ fdisk -l panda-raring_developer_20130922-471.img Disk panda-raring_developer_20130922-471.img: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x00000000 Device Boot Start End Blocks Id System panda-raring_developer_20130922-471.img1 * 63 106494 53216 c W95 FAT32 (LBA) panda-raring_developer_20130922-471.img2 106496 2097151 995328 83 Linux
Now you can see the partitions and the size information. For mounting you need the start offset of the second partition (106496) multiplied with the block size (512).
# mkdir ubuntu-arm/ # mount -o loop,offset=$[start_offset*block_size] panda-raring_developer_20130922-471.img ubuntu-arm/
Another important thing is to put your emulator into the mounted directory.
# cp /usr/bin/qemu-arm-static ubuntu-arm/usr/bin/
In ubuntu-arm/ you should have the whole Arm root directory now.
The Linaro images usually not so big (1GB). Sometimes this image size can be too small. In that case there is a way to expand the image. I will show you how.
First, make a 1GB size file with a program called dd:
$ dd if=/dev/zero of=expand_tmp bs=1M count=1000
If your image is mounted then umount it and add the expand_tmp content (which are just zeros) to the end of your Linaro image:
$ cat expand_tmp >> linaro.img
Now your image is bigger by 1GB, but the second partition size is the same, thus you need to re-size it.
Mount again the second partition of the image:
# mount -o loop,offset=$[start_offset*block_size] linaro.img ubuntu-arm/
With the help of the df program, you can check which /dev/loop device you need to resize.
Resizing the partition:
# resize2fs -f /dev/loop0
The partition is now bigger by 1GB. You can repeat the process whenever you run out of space in your image.
If you want to chroot to your new Arm root directory, you have to prepare it first. You need to mount some important directories from your host system root directory.
Mounting directories:
# mount --bind /proc ubuntu-arm/proc # mount --bind /tmp ubuntu-arm/tmp # mount --bind /sys ubuntu-arm/sys # mount --bind /dev ubuntu-arm/dev # mount --bind /dev/pts ubuntu-arm/dev/pts
You can also mount your host system /home directory and work from there, that way you can keep your Linaro image small and movable.
# mount --bind /home ubuntu-arm/home
The default user in Linaro images is linaro. You can use that or your host system's user, when you chroot in. Using the linaro user is straightforward so I describe how to use your host user. In the chroot you have to change the linaro username to your username in ubuntu-arm/etc/passwd and copy the line with your username in /etc/shadow to ubuntu-arm/etc/shadow and comment out the linaro one. This way you can use your host system username in the chroot.
It is good to know, whether you're in or outside of the chrooot, in order to make it explicit you can modify your prompt to always display it.
Set a name to the chroot:
# echo ubuntu-arm > ubuntu-arm/etc/debian_chroot
You may also want to have network connection in your chroot.
Copy your resolv.conf to networking:
# cp /etc/resolv.conf ubuntu-arm/etc/
After these steps the mounted root directory is ready to use. Let's chroot in it!
# chroot ubuntu-arm/
If you are lucky and you didn't make any mistake you will get a root shell in your Arm root directory. You may set locales to avoid some warning messages after installing new packages. Set locales:
(ubuntu-arm) # locale-gen en_US en_US.UTF-8 (ubuntu-arm) # dpkg-reconfigure locales
Now you can install packages with apt-get and enjoy the brand new Arm environment.
Most probably you will restart your computer for any reason sometimes. If you want to use your chroot again there are some tasks that you need to do. You can do it by hand or you can make a script for it, it's up to you.
Setup Binfmt again:
# mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc # echo ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm-static:' > /proc/sys/fs/binfmt_misc/register
Mount the Linaro image:
Mount the host directories in it:
Mount your /home in it:
After these you can use it again.
Get into the chroot and install the packages what you need.
# chroot ubuntu-arm/ (ubuntu-arm) # apt-get install g++
Change from root mode to your host user.
(ubuntu-arm) # su your_host_username (ubuntu-arm) $
Compile and run a HelloWorld program.
(ubuntu-arm) $ g++ hello.cpp -o hello (ubuntu-arm) $ ./hello Hello! I'm running on Arm! \o/
Now you can start to compile your favorite desktop project in this Arm environment. If you have any questions, you're welcome to comment or contact me.
Dear Gabor,
Thank you very much for the above content, I am most likely going to try this for my students.
I have a question though: for our students we use Lego Mindstorms EV3 kits. This kit has a ARM 9 Processor, 300 MHz. 4 input ports for data-acquisition and 4 ouput ports. The target of this part of the study is that the students learn to program in C. We want them to do this from a Linux OS and we currently use Debian 10. I have followed multiple tutorials / descriptions on the installation of the proper toolchain (from Code Sourcery) and Eclipse with the Eclipse plug-in for the EV3, but unfortunately no luck compiling so far. This is not an unknown phenomenon for me: in my experience one should reserve about 2-3 weeks for trial and error to get the open source stuff working.
I would now like to try your above how-to to implement an ARM 9 environment and compile the EV3 C programs that the students will have to create in this environment. This way I would like to create a "pre-installed" VM for the students so they don't have to waste time on trial and error installation of software but focus on C programming. Do you think your method above will work for the EV3?
Many thanks in advance for your reply and thanks anyhow for the above.
Best regards,
Richard
PS. For my personal development: can you recommend a book and/or course that I can follow to learn how to (successfully) implement a toolchain?
Hi,by following the above steps i have successfully entered into arm environment,but when i am trying to install packages using apt-get install i am getting dpkg no space left error.you have mentioned to repeat the process whenever we run out of space,but what is the process to be repeated i am not clear.Can you please help me to free the space.
(Reading database ... 48146 files and directories currently installed.)Removing gettext-base (0.18.3.1-1ubuntu2) ...dpkg: unrecoverable fatal error, aborting: unable to flush /var/lib/dpkg/updates/tmp.i after padding: No space left on deviceE: Sub-process /usr/bin/dpkg returned an error code (2)
Removing gettext-base (0.18.3.1-1ubuntu2) ...dpkg: unrecoverable fatal error, aborting:unable to flush /var/lib/dpkg/updates/tmp.i after padding: No space left on deviceE: Sub-process /usr/bin/dpkg returned an error code (2)
I realize I am commenting on something written a couple years ago, and I am doing it anyway. Thank you for this. Really. I am not a programmer, but I have used linux my entire adult life and find I greatly benefit from information like this. I've compiled my own entire system from source many times because there are good instructions on the internet that teach how to do it. But figuring out the next step on how to CROSS-compile software seems to be some locked mystery that nobody wants anybody to understand. There are plenty of guides out there on how to do it up to the point of writing your own "Hello World" program and cross-compiling that. But without even trying I knew I could not use just those instructions to compile anything that builds against other libraries.
With the huge, rich, and diverse open-source software library out there combined with all the hundreds, if not thousands, of possible arm-based devices, there is a pretty endless number of combinations/reasons for a non-programmer to want to cross-compile software for an arm device. So why this is the only write-up I can find on the whole friggin' internet that gives instructions that can help a person to actually do that, I don't know. The only reason I found it is because I found myself thinking "I can't see how you could actually build against existing libraries without a chroot" while reading other, rather incomplete cross-compile how-tos. So I searched "cross-compile using chroot" and here I arrived. There may be other ways to do it, but it is not included in what other people write.
The instructions are clear. They make sense. After reading it I do not feel confused and it does not seem like there is some giant hole in the logic ensuring it will never work for what I want to do. Again, a thousand times thank you.