Profiling Android with the HiKey 960

Recently, I reviewed the procedure for system profiling and software tracing of Linux running on the Raspberry Pi 3 using Arm DS-5 Streamline. Similar profiling can be done for systems running Android. While the Raspberry Pi 3 is a great educational board for learning Linux, something different is needed for Android. The HiKey 960 board with a HiSilicon Hi3660 SoC is similar to many mobile devices on the market today, runs the latest Android Oreo, and contains the Cortex-A73, Cortex-A53, and Mali-G71 from Arm. The HiKey 960 is affordable and fully supported by the Android AOSP (Android Open Source Project).

There are various articles and videos about setting up the HiKey 960 so I won’t review the details. If you are interested just search for "HiKey 960 unboxing". Besides the board and the power supply, an HDMI monitor, USB keyboard and mouse, and a USB Type-C cable for flashing the software images are required.

The first step is to connect the HDMI monitor, keyboard, mouse, and power supply to start up Android. The installed version will be out of date, but starting it up is a good test to make sure the monitor, keyboard, and mouse work fine.

When Android starts, it’s a bit of a mystery what do with it. Wireless can be connected, but it has only a rudimentary WebView browser (which cannot download any files), no terminal, and no file manager app. This makes it difficult install any other applications.

There are two ways to install apps from places like APKMirror. One way is to download the .apk on another machine, put it on a micro SD card, put the card in the HiKey 960, and then use Storage menu in the Android settings to get to the SD card. Installing an apk file requires enabling apps from unknown sources in the Security settings. Using this technique, it’s possible to install a few useful apps such as a browser, terminal, and file manager. A second way is to read on and connect adb. With adb, apk files can be installed using the “adb install” command. The initial setup of adb may take longer, but it’s required to get maximum benefit from Streamline anyway and is perfectly fine for sideloading apps.

Let’s get started with the information to enable all features of Streamline on the HiKey 960.

Flashing the latest AOSP

The Android SDK Platform Tools are needed to setup Streamline on the HiKey 960. Download and install the version for the host machine to be used. The Platform Tools provide fastboot and adb. Fastboot is used to flash the software images and adb will connect to the board from the host machine.

To get up to speed with the latest software, download the current AOSP release from Linaro. The file to update everything is the one with “factory” in the name, such as hikey960-linaro-2017.08.23-factory-576c88f9.zip. The builds are numbered and directory named latest/ always points to the most recent build.

Extract the .zip file using unzip on Linux or a suitable program on Windows. It’s helpful to add the directory with fastboot and adb to the PATH environment variable.

Flashing the software is done using the fastboot utility. Official documentation is provided at 96Boards.

To prep for the software update:

• Power down the board
• Move switch 3 to the ON position (push up if you are looking at the numbers right side up)
• Connect a USB Type-C cable from the host machine to the HiKey 960 USB type C connector
• Power on the board
• Run the command “fastboot devices” to see if the board is visible.

If the board is visible the output will be a number and “fastboot”:

447786182000000         fastboot

This is the picture for the default settings, to boot Android. To use fastboot move the top switch in the picture (Ext boot) to the left.

The easiest way to flash the software is from a Linux host using the flash-all.sh script. It assumes fastboot is in the PATH. Although the documentation says only Linux is supported for flashing, Windows probably also works fine if the same commands are executed.

Once the updates are complete:

• Power-off the board
• Move switch 3 (Ext boot) back to the OFF position (as shown in the picture above)
• Remove the USB Type-C connection
• Power-on the board and boot Android

One quirk about the HiKey 960 is that the USB keyboard and mouse will not work when the USB Type-C cable is connected so it must be disconnected to use the keyboard and mouse.

Setting up adb

The Android Debug Bridge (adb) is a must for any development with Android. It is included in the Android SDK Platform Tools. The commands below assume adb is installed and in the PATH on either Linux or Windows.

First, make sure the HiKey 960 is in developer mode. Tap 7 times on the Build number in the configuration Settings -> About phone. This will also enable the board to stay alive when there is no screen activity and allow adb to connect.

To connect adb to the HiKey 960, connect the USB Type-C cable between the host and the board and use “adb devices” to confirm the board is visible. The first time some extra messages about starting the daemon on the host machine will display. This is on the first time only. If the host machine is Linux, sudo may be needed for permission to access the USB device for adb. The important output is the line with the number and “device”. This confirms the board is found by adb.

$ sudo adb devices
List of devices attached
* daemon not running. starting it now at tcp:5037 *
* daemon started successfully *
447786182000000	device 

To avoid the problem with using USB Type-C for adb and for the keyboard and mouse, it’s helpful to connect adb over wireless.

This can be done with the USB Type-C cable connected and then the cable can be removed and a wireless link for adb can be used; assuming the host machine and the HiKey 960 board are on the same network. This enables adb and the keyboard/mouse at the same time.

The commands with $ are on the host machine and the # are on the target via the adb shell. Most instructions will miss the commands on the target which I found are required for HiKey 960.

With the USB Type-C cable connected:

$ sudo adb root
$ sudo adb shell
# setprop service.adb.tcp.port 5555
# stop adbd

This may disconnect adb since it is stopped! If needed go to the Developer options in the Android configuration using the HDMI monitor and slide the USB debugging off and then back on. This should restart adb with the new tcp configuration.

If not disconnected restart adbd:

# start adbd
# exit
$ sudo adb  tcpip 5555

Remove the cable, find the IP address of the HiKey board in the configuration About phone -> Status or type ifconfig in a terminal app on the HDMI monitor, and connect over wireless. For my network I did:

$ sudo adb  connect 192.168.1.63:5555

When necessary come back to this section follow the instructions to connect adb again.

Streamline has three modes of operation:

• Gator Android application
• Gator daemon as root
• Gator daemon as root with Gator kernel driver

Streamline Gator Android application

The quickest way to do something with Streamline is to install the Android apk from the DS-5 installation.

$ adb install $DS5_HOME/sw/streamline/android/streamline-daemon-app.apk

Running the Streamline Android app shows many warnings about security and permission on Android 7+ and many other things which cannot be done.

The net result is the only thing that can be traced with Android O and the Streamline Android application is the gator daemon itself! This is not much help for profiling an application or overall system performance. Continue reading to learn how to overcome this by running the gator daemon as root and use the gator driver to get the most out of Streamline.

Connect Streamline

To just practice how to connect Streamline anyway, setup the path to ADB in the Streamline preferences. Browse to the adb.exe on Windows and set the ADB Path:

Connect to the HiKey 960 board (eyeball) and start a trace by clicking on the red dot.

The result is the profile information for the gator daemon! The screenshot shows the Android application and the underlying libgatord.so

To improve Streamline functionality the gator daemon (running as root) and driver must be used. The gator daemon is a userspace application which can be built using the Android NDK and the gator driver is a Linux kernel driver. First, let’s try with just the daemon and see what is possible.

Building the gator daemon for Streamline

These instructions to build the gator daemon are for a Linux host. A similar flow may be possible on Windows hosts, but I have not tried it.

Install the Android NDK by downloading the zip file from the download area.

Install the NDK by unzipping the file and add the new directory to the PATH environment variable. Then, get the gator source code from github:

$ unzip ~/Downloads/android-ndk-r15c-linux-x86_64.zip
$ export PATH=$PATH:~/android-ndk-r15c
$ git clone https://github.com/ARM-software/gator.git

As covered in the gator daemon instructions for Android, move the daemon/ directory to jni/ for the NDK build system to work.

$ cd gator
$ mv daemon jni

Edit jni/Application.mk to adjust the NDK version and select a 64-bit gatord. Two lines need to be changed so they read:

APP_ABI := arm64-v8a

NDK_TOOLCHAIN_VERSION := 4.9

Now, start the build of the gatord

$ ndk-build

If all goes well, there will be a gatord at libs/arm64-v8a/gatord.

Push the new gatord to /system on the HiKey 960 using adb. The remount command will make /system writable. Without this command the system directory is read-only and no files can be changed.

$ sudo adb remount
$ sudo adb push ~/gator/libs/arm64-v8a/gatord  /system

Now connect and start the gator daemon as root. If needed loop back up to the adb section and review the connection instructions.

$ sudo adb root
$ sudo adb shell
# cd /system
# chmod +x gatord
# ./gatord &

Connecting Streamline with gatord running as root provides access to CPU counters and Linux profiling, but not the Mali G-71 information.

A bunch of warning are shown related to Mali which can be fixed by adding the gator driver to the system.

To get all the system profiling features of Streamline including the Mali GPU, the gator driver must be added to the Linux kernel.

Building the gator driver

Adding the gator driver is a challenge because it’s not possible to load kernel models into the running Android O kernel.

The file /proc/sys/kernel/module_disabled contains a 1 and there is no way to write a 0 to it. This prohibits the use of kernel modules.

This leaves two options:

• Change and rebuild the kernel to allow loadable modules
• Statically compile the gator driver into a new kernel

Both options require the kernel to be recompiled so the second one is probably preferred since it eliminates the extra step of loading the gator module. To maintain the control of unloading the gator module use the first method.

 Either way, the kernel needs to be rebuilt so let's get started.

Rebuilding the kernel with gator

The kernel source tree is available from android.googlesource.com

There are branches for the 4.4  and 4.9 kernels. The Linaro images used at the start of the article use the 4.9 kernel so that is the branch that should be used.

$ git clone https://android.googlesource.com/kernel/hikey-linaro
$ cd hikey-linaro/
$ git checkout -b android-hikey-linaro-4.9 origin/android-hikey-linaro-4.9
$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 hikey960_defconfig

The gator driver source was already retrieved from github for the daemon step. Use the path from that step to copy the driver source.

To add the gator driver into the kernel:

$ cd drivers
$ mkdir gator
$ cp -r /path/to/gator/driver/* gator

Edit the Makefile in the kernel drivers folder and add this to the end:

obj-$(CONFIG_GATOR)     += gator/

Edit Kconfig in the kernel drivers/ directory and add this before the last endmenu:

source "drivers/gator/Kconfig"

The gator driver can now be enabled using menuconfig. It’s in the Device Drivers menu and select the * to include it in the kernel. Select M to build the loadable gator module. If the * is selected then the driver will be included in the kernel and there is nothing else to do except build the kernel. If M is selected then one more change is needed to enable modules to be loadable in the new kernel.

$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 menuconfig

If the M was selected for the Gator module, then module loading can be added by changing the source code to set the value of “modules_disabled” to zero in the kernel source code at kernel/sysctl.c  The last two lines change from &one to &zero as shown below:

                 .procname       = "modules_disabled",
                .data           = &modules_disabled,
                .maxlen         = sizeof(int),
                .mode           = 0644,
                /* only handle a transition from default "0" to "1" */
                .proc_handler   = proc_dointvec_minmax,
                .extra1         = &zero,
                .extra2         = &zero,

To load modules the kernel configuration also needs to select “Enable loadable module support” on the top level configuration screen:

In all cases, continue the kernel build using make. Adjust the cross-compiler to match the compiler to be used.

$ make CROSS_COMPILE=aarch64-linux-gnu- ARCH=arm64 -j 8

The two key outputs from the kernel build are the device tree and the kernel image. They end up in the following locations when the kernel build completes.

• Device tree: arch/arm64/boot/dts/hisilicon/hi3660-hikey960.dtb
• Kernel: arch/arm64/boot/Image.gz

The trick is how to take these two files and create new boot.img and dt.img files to flash using fastboot.

The first missing piece is the ramdisk from the current (original) boot.img

The process to create a new boot.img is to extract the current ramdisk from the old boot.img, combine it with the new kernel Image.gz and re-pack them into a new boot.img.

To extract the ramdisk use the split_boot Perl script found in bootimg_tools_7.8.13.zip

Download and extract the zip file then:

$ chmod +x split_boot  
$ ./split_boot boot.img 
Page size: 2048 (0x00000800)
Kernel size: 8538780 (0x00824a9c)
Ramdisk size: 1989419 (0x001e5b2b)
Second size: 0 (0x00000000)
Board name: 
Command line: 'androidboot.hardware=hikey960 console=ttyFIQ0 androidboot.console=ttyFIQ0 firmware_class.path=/system/etc/firmware loglevel=15 buildvariant=userdebug'
Base address: (0x079fff00)

Writing boot/boot.img-kernel ... complete.
Writing boot/boot.img-ramdisk.cpio.gz ... complete.
Unpacking ramdisk... complete.

$ cp boot/boot.img-ramdisk.cpio.gz  ./ramdisk.img

Now, create new boot and device tree images by using the mkbootimg and the mkdtimg from github:

$ git clone https://github.com/96boards-hikey/tools-images-hikey960.git

Get the utilities from tools-images-hikey960/build-from-source

$ mkbootimg --kernel Image.gz --ramdisk ramdisk.img --cmdline "loglevel=15 androidboot.hardware=hikey960 androidboot.selinux=permissive firmware_class.path=/system/etc/firmware init=/init" --base 0x0 --tags-addr 0x07A00000 --kernel_offset 0x00080000 --ramdisk_offset 0x07c00000 --os_version 7.0 --os_patch_level 2016-08-05 --output new-boot.img
$ mkdtimg -d hi3660-hikey960.dtb -s 2048 -c -o new-dt.img

To update the kernel use fastboot to flash the 2 new images using the same procedure as described above to move the switch, run fastboot, put the switch back, and restart.

$ fastboot flash dts new-dt.img
$ fastboot flash boot new-boot.img

With the new kernel installed, the board restarted, restart the gatord and connect Streamline.

Now, all features of Streamline can be used. This includes CPU counters, Mali counters, and full process tracing.

Summary

Setting up full Streamline functionality with the HiKey 960 board is a bit more complex than the Raspberry Pi 3, but the solid support of the Android Open Source Project (AOSP) and the work by the Linaro and 96Boards communities make it a great development board for those who want to dig into the latest version of Android on recent, high-performance hardware which is similar to current mobile devices. The full functionality is provided without the need to compile the full Android AOSP userspace. For those who have a host machine with 150 Mb of disk space and 16 Gb of RAM AOSP can be compiled using the AOSP instructions, but the steps here make it possible to use Streamline without performing the full AOSP build procedure.

Give Streamline a try by visiting to the DS-5 download page using the button below and requesting a trial license:

[CTAToken URL="https://developer.arm.com/products/software-development-tools/ds-5-development-studio/downloads" target="_blank" text="DS-5 Downloads" class ="green"]