Bringing up Streamline in the Android Developer Toolchain (SDK and NDK) in order to do system analysis for your Android app

This blog has been written before but quite a few things have improved in the ease of use and breadth of support of Streamline use on Android in the past few years. For starters, Mac OS X is well supported. Now all three major development platforms (Linux, Windows and Mac) have the ability to run DS-5 Community Edition debugger (gdb) and Streamline with the ADT Eclipse tools from Google as an add-on or pre-packaged as DS-5 CE for Windows and Linux from ARM with ADT as an add-on. Also, and most welcome, is the new Gator driver. The component of Streamline that runs in Android to collect OS and processor counters used to require both a kernel module and a driver daemon. Compiling and flashing any module could be complicated depending on the availability of your Android platform kernel headers. That requirement has been removed and now the Gator driver will run as root on many devices. This July (7/2014), an updated version of gatord in the DS-5 CE 5.19 will be released that greatly expands the kernel versions supported (beyond the 3.12 kernel version supported in the current DS-5 5.18 release). Finally, I’ve found some erroneous and dated info in some blogs that claim to be up to date to DS-5 5.18 and even the yet to be released 5.19. I’ll try to correct that here and support this blog entry.

Streamline is a powerful system analysis tool that will help you speed up your code, reduce energy footprint and balance system resources. The free version in the Community Edition of DS-5 lets you view CPU and OS counters in a powerful graphical view; CPU and GPU activity, cache hits and misses and visibility down in to individual threads and modules. You can find code that is blocking or could be optimized by multithreading or refactoring in NEON or the GPU. Check out more features on the optimize site.

Getting Started:

As of this writing the Android SDK Manager is Revision 22.6.4 bundled in the latest SDK for Mac, adt-bundle-mac-x86_64-20140321. The SDK is available at the Android Developer Site. The Native Development Kit (NDK) is revision 9d. Download both of these for your appropriate platform. I’m downloading the Mac OS X 64-bit versions for this guide but these instructions should work for Windows and Linux just as easily.

Once you unpack these tools, you should add some executable paths to your platform if you plan on using the terminal for anything like the Android Debug tool (adb). It is now possible to use all of the tools from within Eclipse without adjusting your executable paths but for some of us old-schoolers who are wedded to the CLI, I drop my NDK folder in to the SDK folder and put that folder in my Mac’s /Applications directory. You can place them wherever you like on most platforms though. I then added these to my ~/.bashrc

export PATH=$PATH:/Applications/adt-bundle-mac-x86_64-20140321/sdk/platform-tools

export PATH=$PATH:/Applications/adt-bundle-mac-x86_64-20140321/sdk/tools

export PATH=$PATH:/Applications/adt-bundle-mac-x86_64-20140321/android-ndk-r9d

You should now be able to launch common Android tools from your command line:

> which ndk-build

/Applications/adt-bundle-mac-x86_64-20140321/android-ndk-r9d/ndk-build

> which fastboot

/Applications/adt-bundle-mac-x86_64-20140321/sdk/platform-tools/fastboot

> which adb

/Applications/adt-bundle-mac-x86_64-20140321/sdk/platform-tools/adb

> which android

/Applications/adt-bundle-mac-x86_64-20140321/sdk/tools/android

You can Launch the Android SDK Manager from Eclipse in the “Window” menu or via the command line by typing:

> android

From there, you can update your current SDK, install older APIs, build-tools, platform tools and in “Extras”, the Android Support Library for compatibility with older APIs.

When you run Eclipse (ADT) for the first time or change versions, you may have to tell it where to find the SDK. The Preferences dialog box is found on Macs via the ADT->Preferences menu, sub heading Android.

Setting up a demo app to analyze (if you don’t have your own app):

You probably have your own library or application you want to perform system analysis on but just in case you’re checking out the tool, I’ll step through setting up an app that is near and dear to me, ProjectNe10. You can grab the master branch archive from GitHub. For this tool demo, I’ve created a directory /workspace and unzipped the Ne10 archive inside that folder. ProjectNe10 requires the cmake utility. Fortunately, there is a Homebrew solution to install cmake from the command line:

brew install cmake

If you don’t have brew installed, install it. You’ll use it in the future, I promise. You can also just download the binary for any platform from cmake.

Now we can build the Ne10 library from the command line:

Set these to your particular paths:

export NE10PATH=/workspace/projectNe10

export ANDROID_NDK=/Applications/adt-bundle-mac-x86_64-20140321/android-ndk-r9d

Then:

cd $NE10PATH

mkdir build && cd build

cmake -DCMAKE_TOOLCHAIN_FILE=../android/android_config.cmake ..

make

make install

That make install line will copy libNE10_test_demo.so to your /workspace/projectNe10/android/NE10Demo equivalent. Now you can go to the File->Import menu in Eclipse and import an existing Android code base in to your workspace.

If all goes well, you should be able to connect your ARM based Android Device (in my case, a Nexus 5 running Android 4.4.4 to match the current SDK at the time of this writing) and run this app from the Run menu as an Android app. As a sanity check, you should run adb devices from the command line to verify you can see your device. This app will iterate through every function in the ProjectNe10 library with both C and NEON implementations. One of the implementations should be faster. I’ll give you a hint. It is the NEON implementation.

Installing DS-5 Community Edition (Free Eclipse Plugin with enhanced ARM debug and system analysis):

Start Eclipse and go to the menu Help->Install New Software.... Click on “Add...”, and paste http://tools.arm.com/eclipse in the location text box, then click OK. Select ARM DS-5 Community Edition, as shown on the screenshot below, and click Next. Eclipse will compute the dependencies of the DS-5 CE plug-ins.

Click Next again. Read the license agreements and if you accept, hit Finish. After the install is complete, ADT will ask you to reload.

A license dialog should popup if this is a fresh install. Select "Install Community edition license" and click "Continue".

If there was no popup license message go to Help->Generate community edition license, and click "Finish".

Congratulations, you now have ARM DS-5 CE installed with its enhanced and easy to use debugger which you can use to debug Android NDK apps and libraries with the steps in this guide. You also have Streamline; a powerful system analysis tool which we’ll cover in the next section.

Using Streamline and gator to analyze Android apps and the entire system

Before you can gather data for system analysis, you have to install a data collecting driver (daemon) in Android. Gatord will gather processor and kernel counters on the Android device and stream them over to your host machine. It must run as root to do this. Any device with an unlocked boot loader is very simple to root, you usually just flash a custom recovery tool like TWRP and install SuperSU. If you have a locked bootloader, you’ll have to use a device exploit so I can’t recommend this or help you but your favorite search engine might… This is a minor inconvenience now as older versions required a kernel module (gator.ko) which needed to be compiled against your particular device’s kernel headers. Now that Android security terms to pass Android CTS disallow kernel modules, you’d have to compile in to the kernel and flash it. Fortunately the new gatord will expand its kernel version support significantly in July.

First, build gatord. Go to the menu Help->ARM Extras… this will open up a folder with several goodies in it.

I’m going to build this from the command line so fire up your favorite terminal and cd in to this directory. The easiest way in the Mac terminal app is to type “cd ” and dragging the gator folder in to the terminal window. OS X will fill in the path, then:

cd daemon-src

tar zxf gator-daemon.tar.gz

mv gator-daemon jni

cd jni

ndk-build

These steps should unzip the gatord source, and build it for Android (dynamically linked) with the output in ../libs/armeabi/gatord. Copy this binary to your Android device with your favorite method, AirDroid, scp, Droid NAS or very simply:

adb push ../libs/armeabi/gatord /sdcard/gatord

This, of course, assumes you’ve enabled developer options and debugging on your device. “On Android 4.2 and newer, Developer options is hidden by default. To make it available, go to Settings > About phone and tap Build number seven times. Return to the previous screen to find Developer options. In Developer options click USB debugging. If this is a new device, you may have to approve the debug link security the first time you try to use adb. You can also do this with an ARM based Android Virtual Device (AVD) in the emulator if your physical device is too ‘locked down’ but Streamline system data won’t be as useful. You may have to use “mount -o rw,remount rootfs /“ and “chmod 777 /mnt/sdcard” in your AVD to push gatord.

Now, the tricky part, you have to move this binary to an executable location in the filesystem and set executable permissions. The most reliable method I’ve used is ES File Explorer. Go in to the menu, turn on Root Explorer and go to the Mount R/W option, set root “/“ as RW (read/writeable) rather than RO. Then copy and paste gatord in to /system/bin in your Android filesystem. You can also set the permissions to executable in ES File Browser by long pressing on the gatord file, then more->Properties->Permissions->Change. Give the owner any group Execute permission and press Ok.

Back in your host machine terminal you need to set up a pipe for gator to communicate over USB and then get a shell on the device to start it:

adb forward tcp:8080 tcp:8080

adb shell

Now you’ve got a shell on your android device, you can su to root and start gatord. Type:

su

/system/bin/gatord&

The rest is pretty straight forward. Go to the Window->Show View->Other…->DS-5->ARM Streamline Data

Click on the gear button

In the address section, enter “localhost” if you’re streaming the capture data over USB using adb to forward the TCP port. In the Program Images box select the shared library that you want to profile (add ELF image from workspace).

You can now use the red “Start Capture” button at any time.

Other blogs and tutorials are accurate from this point forward on the features and use of Streamline so I’ll drop a few and let you get to it!

The “CAPTURING DATA AND VIEWING THE ARM STREAMLINE REPORT” section of this blog is accurate.

Events based sampling video, analyzing CPU and GPU performance and customizing charts on YouTube.