More blogs in Arm Community blogs
Tell us what you think
Tags
Actions

Profile firmware with Performance Monitor Unit (PMU) in Armv8-A CPU

Jiaming Guo
10 minute read time.

The Performance Monitor Unit (PMU) in Armv8-A CPU provides hardware-level performance monitoring and profiling capabilities. The PMU collects hardware event counts through counters. The counters include cycle counter and event counters. You can configure:

  • Each event counter to count specified hardware events.
  • Each counter to collect hardware events from workloads at various CPU Exception levels and states.

Tools like perf use PMU to profile Linux applications running on Armv8-A CPU. However, there is no common way to use PMU to profile the firmware, because it runs in the bare-metal environment.

To profile the firmware with PMU, a straightforward approach is to add PMU support in the form of a library. In this blog post, we provide a PMU library as a reference implementation. This library has been verified in the platform based on Armv8.0-A CPU. We also describe how to add PMU support in the firmware and profile the firmware with PMU. The Trusted Firmware-A (TF-A) and U-Boot are used as example firmware.

Download the PMU Library

The table below lists components of the PMU library.

File name Description
arch.h System registers bit field definitions
arch_helpers.h Inline functions for system registers access (MRS/MSR)
armv8_pmuv3_events.h Structures for PMU event definitions
armv8_pmuv3_fn.c Armv8-A CPU PMU profiling functions implementation
armv8_pmuv3_fn.h Profiling interfaces definitions
jevents.py Python script to get supporting PMU events for a specified CPU and generate armv8_pmuv3_events.c

Stage 1: Add PMU support to the firmware

Use the following command to unzip all the files of the PMU library to the path lib/pmu in the firmware. 

Depending on the firmware you are working with, you need to make specific changes to the Makefile file as follows.

TF-A

Modify the existing Makefile file located in the root path of TF-A as follows.

U-Boot

Modify the existing Makefile file located in the root path of U-Boot as follows.

Create a Makefile file in the same directory with the PMU library. Fill the file as follows.

Now, you can rebuild the firmware. If you build it successfully, go to stage 2.

Stage 2: Select PMU events for profiling

First, get the supporting PMU events for the Armv8-A CPU on which the firmware runs. See the CPU Technical Reference Manual (TRM) for the PMU events.

In the PMU library, you can also use the Python script named jevents.py with the following steps.

Get the supporting PMU events for the specified CPU

The GitHub repository Machine-readable data maintains PMU events supported by each Arm CPU. It uses JSON format to store information such as event mnemonic and event number of the PMU events. This information is consistent with that in the CPU TRM.

The Python script named jevents.py in the PMU library clones the repository, converts the JSON format to C format, and generates the file named armv8_pmuv3_events.c. You can use it as follows.

First, use the following command to list all the names of the supporting CPUs for converting.

Then, use the following command to specify the name of the CPU for converting. This command will generate the file named armv8_pmuv3_events.c. The file contains two types of arrays. One is for the supporting PMU events of the specified CPU. The other is for the selected PMU events for profiling.

An example of the armv8_pmuv3_events.c file generated by jevents.py is as follows. We specify the cpu name as cortex-a53. The array named pmu_events_map contains all the supporting PMU events of Cortex-A53. It is the same with the Cortex-A53 TRM. You can select the PMU events from here and put the selected ones into the array named evt_select.

Select PMU events for profiling

In the initial profiling, normally you need to select a wide range of PMU events. This will get you a comprehensive understanding of the code to profile.

For Armv8-A CPUs, the available event counters per CPU are limited. You can refer to the CPU TRM for the number. If the required number of PMU events exceeds the available counters, you can use the following method.

  1. Divide all the PMU events to select into groups. In each group, the number of the events must not exceed the available counters of the PMU.
  2. Create multiple pmu_event_selected structure arrays and put each group into one array.
  3. Profile the same code multiple times, select one array at a time, and repeat the process until all events are measured.

For example, we create two pmu_event_selected structure arrays named evt_select_cache and evt_select_wlc in the armv8_pmuv3_events.c file. For Cortex-A53, one cycle counter and six event counters are available. The number of PMU events in each array does not exceed this. Also, within each array, the PMU events are related to make sure the profiling data is comparable.

We use these two arrays for the following profiling examples.

Stage 3: Profile the firmware with the PMU

Now, you can profile the firmware with the PMU. Use the following procedure for the specific code in the firmware that you want to profile.

  1. Place start profiling point before the code. This configures, enables, and reads the current value of each PMU counter as pre-profiling value.
  2. Place stop profiling point after the code. This disables and reads each PMU counter value as post-profiling value.
  3. Rebuild the firmware and run the code again. Calculate the difference between the pre-profiling and post-profiling value of each PMU counter. This collects the statistical profiling result.

In the PMU library, armv8_pmuv3_fn.c provide a reference implementation.

Place profiling points in the firmware

You can simply add two functions named pmuv3_startProfiling and pmuv3_stopProfiling between the code that you want to profile. For each profiling, pass one pmu_event_selected structure array as the parameter.

Depending on the number of PMU events you want to profile with, you need to profile once or multiple times. Here are two profiling examples.

Example one

To profile the function enable_mmu_el3 in TF-A with focusing on cache behavior, you can only select the array named evt_select_cache for a single profiling.

Example two

To find out the crc32 workload characterization in U-Boot, you can select the arrays named evt_select_cache and evt_select_wlc. In this way, you need to profile the same code multiple times to collect all the profiling data.

As the do_mem_crc function is called by a command, you can program the firmware once and repeat the profiling by entering the same command multiple times.

Understand the implementation of profiling with the PMU

The function startProfiling does the following things.

  1. Performs necessary initialization for the PMU. It checks whether the number of event counters supported by the CPU's PMU is suitable for the selected events. Then, it sets the MDCR_EL2 or MDCR_EL3 to enable the PMU profiling at the current Exception level. This is because firmware typically runs at EL2/EL3 or in the secure state where profiling is prohibited to prevent information leakage.
  2. Configures each PMU counter to profile at the current Exception level and for the selected PMU events.
  3. Enables each PMU counter.
  4. Reads the current value of each PMU counter as the pre-profiling value.

The function stopProfiling does the following things.

  1. Disables each PMU counter.
  2. Reads the current value of each PMU counter as the post-profiling value.
  3. Dumps the result of this profiling.
  4. Performs necessary deinitialization for the PMU. This disables the PMU working at the EL2/EL3 or in the secure state.

Collect the statistical profiling result

Now you can rebuild the firmware and run it again to collect the statistical profiling result.

For the examples mentioned in the previous part, we perform the profiling on the Juno r2 platform. The output of the profiling is as follows.

Example one

As you can see from the output, it records the pre-profiling and post-profiling value of each selected PMU events and cycles in this profiling. Also, it calculates the differences, and records them in the column named DELTA.

For Cortex-A53 CPU, the PMU event L1D_TLB_REFILL is included in the count for L1D_CACHE_REFILL. Thus, you might observe that the counting value for L1D_CACHE_REFILL is greater than that of L1D_CACHE.

Example two

As you can see above, the results of cycles for the two profiling sessions are almost the same.

From the first profiling result, you can refer to the Arm Architecture Reference Manual to abstract some meaningful metrics as follows.

Metric Formula Value
Attributable Level 1 data cache refill rate L1D_CACHE_REFILL / L1D_CACHE <1%
Attributable Level 2 unified cache refill rate L2D_CACHE_REFILL / L2D_CACHE 37.7%

From the second profiling result, you can calculate the Instructions Per Cycle (IPC) as follows.

IPC = INST_RETIRED / CYCLES

The IPC of this workload is 0.23. This is caused by many MEM_ACCESS operations.

  • 1 comment
  • 0 members are here
Anonymous
Architectures and Processors blog
0