Annotating ARM Streamline Profiles of Mozilla Browsers from JavaScript

Note: This was originally published on the 14^th of February 2012 at blogs.arm.com.

I have recently been using the ARM Streamline profiler to study the behaviour of Mozilla Mobile Firefox (code-named Fennec) on Android. Streamline is a graphical profiling tool that is provided with ARM's DS-5™ development tool suite. Some time ago, whilst investigating a Fennec performance regression bug using Streamline, I had noticed some unexpected activity on the browser's main process. However, it was not clear whether the activity was some periodic event unrelated to the benchmark (perhaps related to garbage collection) or something triggered by the benchmark itself. The graphical timeline view in Streamline is very good for identifying areas that might benefit from optimization, and for highlighting anomalies and bottlenecks. However, when running large tests or benchmark suites, it is difficult to map anomalies in the profile graph to individual test and benchmark stages.

This article explains how Streamline's annotation feature works, and describes the development of a simple Fennec extension that allows JavaScript code in an instrumented benchmark to annotate profile results. Whilst the extension itself is specific to Fennec, the principle is not, and it should be possible to port the extension to other browsers if required. As mentioned, the extension was inspired by Fennec bug 658074, but my motivation behind writing it was that it could be used for numerous other profiling tasks. For example, it can be used to profile JavaScript-triggered CSS animations or page-layout tasks. HTML5 videos are also controllable using JavaScript, so the control code could be instrumented to annotate a profile when video is started, paused, or even resized and scrolled around the view.

It is important to note that this is the first browser extension I've written. I am not an expert on Fennec extensions, and I won't be describing the boilerplate parts of the extension in any particular detail. If you're looking for a general guide to writing Fennec extensions, I'd recommend starting with Mark Finkle's excellent video tutorials.

Those just looking for a link to the extension can find one here: armgator.xpi

If you're viewing this page with Fennec, you will be able to install the extension directly from the above link. If you're using desktop Firefox and intend to download the file to feed to Fennec manually, you'll need to download it using the context (right-click) menu, otherwise Firefox will try (and fail) to install it. The add-on is configured to support Fennec 4.0b1 and above. Whilst an upper limit on the supported version is usually wise, I have chosen to omit it in this case, partly because I don't want to have to update it for every Fennec release, and partly because it's a development tool used for working on the browser code itself, not a typical user-facing extension.

Note that XPI files are really just ZIP files with a particular directory layout, so you can get the source from the XPI using your ZIP-reading tool of choice.

Development Environment

My target device in this case is a Beagleboard xM, and I'm using the example system image provided with DS-5¹. If you are using a board with no example image, or you want to use a custom system image, you will need to rebuild the Gator kernel module and install the Gator daemon. The Gator kernel module and daemon (along with instructions for building and installing them) are provided with DS-5².

Be aware that you will need root access to your device in order to use Streamline, and you will need to have access to suitable kernel sources so you can build the kernel module.

In order to allow Streamline to talk to the target, I am using an Ethernet connection³. Streamline should work just as well over a wireless connection, in case you are using that, or indeed any other connection that allows TCP traffic.

To let Streamline talk to your target, you'll need to use ADB — the Android Debug Bridge — to set up port forwarding. On your workstation (where you will run Streamline):

$ adb connect [target IP address]
$ adb forward tcp:8080 tcp:8080

You should now be able to connect Streamline to your target using address localhost and port 8080. Refer to the documentation provided with the Streamline distribution for details of how to connect Streamline to a target.

The following image shows a screenshot of the Streamline's Timeline view after running a subset of Mozilla's Kraken benchmark on Fennec. There are some interesting spikes in the L1 data miss and the branch misprediction counters, but it's not clear what is actually going on here. (Of course, if you try this yourself, you may have different performance counters selected and you may see something different.)

Streamline Annotations

Streamline supports a method of annotating the Timeline view. These annotations work similarly to printf, but with timing information built in. Each annotation can have a start and end time and can be shown directly on the timeline view.

Operation Principles and a Quick Test

An application can send an annotation at any time simply by writing a NULL-terminated string into a special file: /dev/gator/annotate. Whilst not particularly useful for real profiling, it is also possible to verify that annotations are working in your development environment using a single shell command on the target (using adb shell).

echo -e 'Hello, Streamline.\0' > /dev/gator/annotate

The echo command above will also send a newline after the message. Normally, this could be surpressed using the -n option, but for some reason that eludes me, it does not seem to be possible to specify both -n and -e to Android's echo. The command above will therefore set the annotation message and then start a second message with a newline. The usual truncating effect of the redirect also doesn't seem to work when writing a subsequent message. In any case, the command is suitable for verifying that Streamline can receive your annotation, and you should be able to get something that looks like this:

Note that the annotations are hidden by default. To show them, click the blue triangle — next to the process name in the list on the lower left — to expand the process details. In this case, because the annotation was issued from the shell, the process name is [sh].

Annotations are shown with both start and end times. An annotation can be ended either by sending a new annotation to replace it, or by setting an empty annotation string (by writing a single NULL character to /dev/gator/annotate).

C Code

The usual usage of annotations is from instrumented programs that you want to study. In most cases, it is likely that you are using C or C++. It is pretty simple to write annotations from C, and the Streamline documentation includes a header file with helpful macros to do the work for you, but the following C snippet will suffice for a quick test:

// Send a Streamline annotation from a C program.
FILE * f = fopen("/dev/gator/annotate", "wb");
fprintf(f, "Hello, Streamline.");  // Write the annotation message.
fputc('\0', f);                    // Write the NULL termination.
fflush(f);                          // Flush the I/O buffer.
fclose(f);

Colours

Finally, it is possible to add colour to annotations. These colours are used as the background colour for the annotations in the timeline view, and they can be very useful for quickly identifying milestones and other interesting points in a profile. The colours are set simply by writing an ASCII escape character (0x1b) at the start of an annotation, followed by a byte for each of the red, green and blue components. As before, the utility header file provided in the Streamline documentation provides macros to make this simple, but the following C snippet provides a quick example, annotating with a dark red background colour:

// Send a coloured Streamline annotation from a C program.
FILE * f = fopen("/dev/gator/annotate", "wb");
fputc('\x1b', f);                  // Escape character.
fputc('\x80', f);                  // Red channel value.
fputc('\x00', f);                  // Green channel value.
fputc('\x00', f);                  // Blue channel value.
fprintf(f, "Hello, Streamline.");  // Write the annotation message.
fputc('\0', f);                    // Write the NULL termination.
fflush(f);                          // Flush the I/O buffer.
fclose(f);

The following screen-capture shows how coloured annotations (from a trivial test application) are displayed in Streamline:

Note that the activity on the [annotate] process is close to 100% (red) because I implemented a crude time delay using a busy-loop.

The Fennec Extension

There are many resources on the web which cover the subject of writing Firefox and Fennec extensions, but it was a new subject to me and the requirements of this extension are perhaps slightly unusual. Specifically, it needs to be able to write both binary and text data to a file on the local system, and it needs to make this functionality available to the JavaScript running in the page content. Any extension that allows client JavaScript to write to the local file system has the potential to expose a major security risk, so some thought must go into the design. Having said that, security is obviously less important for an extension used exclusively for Fennec development work than it would be for a user-facing extension.

Extension Design and Requirements

This extension has to do two things that (at first) did not appear trivial to implement from a JavaScript extension:

• The JavaScript environment presented to the page content needs to be modified. Effectively, to provide an API to JavaScript in a page, the extension needs to add an object (containing some functions) to the content page's global object. This will allow content JavaScript to call into the extension.
• The extension has to write into a file — /dev/gator/annotate — on the local file system. Both binary and text data need to be written: The colour information is in a binary format, whilst the annotation message is just simple text.

I assumed that the above would be easier to achieve from C++ in an XPCOM extension, rather than from JavaScript. However, after some fruitless experimentation followed by a very useful discussion with Mark Finkle (mfinkle) and Ted Mielczarek (ted) on Mozilla's #mobile IRC channel, it became apparent that these requirements are actually fairly simple in a pure JavaScript extension. There are just a couple of catches:

• Fennec runs in two separate processes⁴:

  • org.mozilla.fennec_[...]
    • In my case, this thread was called org.mozilla.fennec_unofficialbecause I built Fennec without the official branding. More recent versions of the Fennec source seem to name it org.mozilla.fennec_[user-name], but it is the same thing. This process is responsible for running the browser chrome, which includes the user interface elements such as the location bar, but not the page content.
  • plugin-container
    • The page content runs and is rendered in a process called plugin-container.

• This process split is done for a few reasons which I won't describe here in detail, but the split is relevant because Android only allows the main process (org.mozilla.fennec_unofficial) to write to the local file system. Because the page content cannot directly work on the main thread, the extension needs to use a message-passing API to pass the annotations from the content process (where the content JavaScript runs) to the main process, which can actually handle the annotation.

• To make an object usable by content JavaScript (in order to provide an API), it has to be added to the page's JavaScript environment. However, it is also necessary to explicitly mark each content-accessible property using a special __exposedProps__ array, which is a member of the new object. This is quite easy to do once you know that you have to do it, and it explains why my initial experiments had failed: Whilst I'd successfully added an object to the content page's global object, I had not made it visible.

Luckily, neither of the above catches actually cause much trouble. There are documented APIs for writing to files from JavaScript, and making extension functions visible to page content is simple using __exposedProps__. The extension's code includes comments explaining all this, but the next section will give a functional overview.

The Structure of the Fennec Extension

The directory structure of my extension looks like this:

armgator.xpi
├── chrome.manifest
├── content/
│  ├── content.js
│  ├── options.xul
│  ├── overlay.js
│  └── overlay.xul
├── defaults/
│  └── preferences/
│      └── prefs.js
├── install.rdf
└── LICENSE

As I mentioned at the start, I'm not going to explain every file in detail. I'm also not going to explain most of the code in detail, as that is really boring and the comments in the code should be sufficient. However, here's a brief summary of what each file is for:

• The implementation of the JavaScript part of the extension is split between overlay.js and content.js, as highlighted in the directory structure diagram. These files form the interesting parts of this particular extension.

• The configuration options for the extension are defined in options.xul, and the default values for each option are set in prefs.js.

Everything else is just boilerplate extension code. Several standard files are omitted simply because they aren't used and would be empty. The following files remain:

• The install.rdf file describes the extension to the browser and specifies several attributes, such as the extension name, as well as which versions of Fennec it supports.

• The chrome.manifest file tells Fennec where to find the extension content.

• The overlay.xul file defines the visual part of the extension. This extension has no visual component (other than the configuration interface), but the browser expects to load a XUL overlay, not a JavaScript file. For this extension, overlay.xul simply pulls in the overlay.js file, where the extension is implemented.

Extension Design

In order to cope with the multi-process design and its restrictions, this extension uses two separate modules which communicate using the message-passing API:

overlay.js

overlay.js contains the code that is run when the extension is loaded. It runs in the context of the browser chrome — in the org.mozilla.fennec_unofficial process — so it can write into local files (and thus send annotations to Gator), but it cannot be accessed directly from content code.

content.js

content.js contains functions that are directly accessible to content code. It cannot directly access the annotation file, but it can use the message-passing API to talk to overlay.js. overlay.js can then send annotations on the behalf of content.js.

The Code

I won't describe the code in detail here, mostly because it's probably more useful to look at it directly. Because the extension is implemented using pure JavaScript, you can simply download the extension and extract it. The extension XPI is a simple ZIP file.

Using the Extension

The extension can be installed in the usual way. Feeding Fennec the XPI is the simplest method. Once installed, Fennec will look pretty much the same. If you navigate to Fennec's 'Add-ons' page, however, you'll see that there is a new extension, with some options:

To test the extension on a device that doesn't have Gator installed, perhaps to verify that it works before installing it on your target device, simply set the annotation file to some empty, local file. The file will be updated when an annotation is sent by content code. You could also enable the alerts feature to get a system alert each time an annotation is sent. The alerts seem to work well when running Fennec on Linux, but not so well on Android because Android's notification area gets rather cluttered once several annotations have been sent.

The extension won't actually do anything unless some JavaScript code on a page tries to send an annotation, of course. The following code is probably the best way to do this, if you're instrumenting a benchmark:

if (window.Gator && window.Gator.annotate) {
    Gator.annotate("Annotation Text");
}

As a quick test, if you're viewing this page with Fennec and you have the Gator add-on installed, click here to start an annotation and click here to to end it. Using Fennec on Linux (running in a window), and with alerts enabled, the annotation sent by the first link looks like this:

Finally, the following screenshot was taken from Streamline after running an annotated profile of a subset of the Kraken benchmark:

¹The example system image provided with DS-5 includes an Android kernel (with pre-built Gator module) and an Android file system (with Gator daemon). In DS-5 release 5.5 (build 966), the example system image is available in [ds5‑installer‑directory]/linux_distributions/beaglexm.zip. Other releases may vary. Refer to the provided READMEs and suchlike for various configuration and build instructions.

²In DS-5 release 5.5 (build 966), the Gator kernel module source is available in [ds5‑install‑directory]/arm/gator/src. Other releases may vary. The Gator daemon is provided in [ds5‑install‑directory]/arm/gator/android/gatord. Refer to the provided READMEs and suchlike for various configuration and build instructions.

³For me, getting the Ethernet connection to work required some fiddling because on my board, for some reason, I have to run netcfg manually to get an IP address from DHCP. Also, the DHCP client sets net.usb0.dns[N] (presumably indicating that the Beagleboard xM's Ethernet chip is on the USB bus), but the system expects to read net.dns[N]. I can get Ethernet networking up and running properly using the following commands (as root):

# netcfg usb0 dhcp
# getprop net.usb0.dns1  // Get first DNS server's IP address.
# getprop net.usb0.dns2  // Get second DNS server's IP address.
# setprop net.dns1 [first DNS server's IP address]
# setprop net.dns2 [second DNS server's IP address]
# netcfg                  // Get the target's IP address (on usb0).

⁴As I understand it, the process split will not exist in the Native UI releases. However, I think there will still be a thread split, and the message-passing mechanism should still function in the same way.