Guide to Set Up Linux kernel Debug Environment on Arm DS for Total Compute Platform

The Linux kernel within the Arm Total Compute platform includes subsystem-specific features that showcase the platform's capabilities. However, because Linux operates at different exception levels and addresses, debugging it with Arm DS can be challenging.

This guide describes the following tasks:

How to set up the Linux Kernel debug connection.
How to disable the Linux Kernel Address Space Layout Randomization.
How to load the Linux Kernel debug symbol file.
How to start the Linux Kernel debug.

This guide assumes that you already set up the debugging environment as described in the Guide to Set Up Debugging Environment for Total Compute Software Stack.

Note: The guidance given in this guide is based on the Total Compute TC2-2023.10.04 code. Future code updates might introduce changes, so the guidance might not apply to all cases.

How to set up Linux Kernel debugging connection

The Guide to Set Up Debugging Environment for Total Compute Software Stack describes how to set up the Total Compute debugging environment in the Arm DS.

There are some considerations to be aware of when setting up the Linux Kernel debugging environment.

When debugging the Linux Kernel, select ARM_Cortex-A520x4 SMP Cluster 0 as your target as shown in the following figure:

How to disable the Linux Kernel Address Space Layout Randomization

Kernel Address-space layout randomization (KASLR) is a well-known technique to make exploits harder by placing various objects at random, rather than fixed, addresses. To make debugging the code more convenient, we hope to keep the address of the code fixed every time it runs, so that the symbol table can be easily loaded during debugging. To achieve this, there are two methods to disable KASLR.

1. To disable it during compilation, you can modify the boot arguments. This can be done by editing the file build-scripts/files/u-boot/tc2/fvp.cfg, as shown follows:

diff --git a/files/u-boot/tc2/fvp.cfg b/files/u-boot/tc2/fvp.cfg
index 91b9e54..8e86577 100644
--- a/files/u-boot/tc2/fvp.cfg
+++ b/files/u-boot/tc2/fvp.cfg
@@ -1,5 +1,5 @@
 CONFIG_TARGET_TOTAL_COMPUTE_FVP=y
-CONFIG_BOOTARGS="console=ttyAMA0 debug user_debug=31 earlycon=pl011,0x2A400000 loglevel=9 androidboot.hardware=total_compute androidboot.boot_devices=1c050000.mmci ip=dhcp androidboot.selinux=permissive allow_mismatched_32bit_el0 systemd.log_level=info"
+CONFIG_BOOTARGS="console=ttyAMA0 debug user_debug=31 earlycon=pl011,0x2A400000 loglevel=9 androidboot.hardware=total_compute androidboot.boot_devices=1c050000.mmci ip=dhcp androidboot.selinux=permissive allow_mismatched_32bit_el0 systemd.log_level=info kaslr.disabled=1"
 CONFIG_BOOTDELAY=1

2. At runtime, you can change the boot arguments in the U-Boot command line, as shown follows:

Hit any key to stop autoboot:  0
TOTAL_COMPUTE# setenv bootargs  ${bootargs} kaslr.disabled=1
TOTAL_COMPUTE#

How to load the Linux Kernel debug symbol file

From version 5.11 of Linux kernel, If the kernel is booted at EL2, it always start with a nVHE init, drop to EL1 to initialise the the kernel.Then upgrade the kernel EL to EL2 if VHE capable, as shown follows:

Depending on the different debugging requirements, it may be necessary to focus on multiple stages of the code during the debugging process. To effectively debug these stages, it is essential to add the corresponding symbol tables for each stage.

1. Load the debug symbol for the booting process at EL2 with MMU disabled

First, you need to find the kernel load address. On the Arm Total Compute platform, the Linux kernel is loaded by U-Boot. You can find the address where the kernel is loaded by checking the U-Boot logs. Refer to the following log as an example:

## Checking Image at a0000000 ...
   FIT image found
   FIT description: Total Compute: fitImage
    Image 0 (kernel)
     Description:  Linux Kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0xa00000c4
     Data Size:    38554112 Bytes = 36.8 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x80200000
     Entry Point:  0x80200000
     Hash algo:    sha1
     Hash value:   32e5e3d98fd89160e6379c7f5287c3ab5060347c

From the log above, it can be found that the kernel is loaded at the address 0x80200000, which corresponds to the address of the symbol table _text in the kernel.
Reference file: arch/arm64/kernel/vmlinux.ld.S

	. = KIMAGE_VADDR;

	.head.text : {
		_text = .;
		HEAD_TEXT
	}

Next, you can find the address of _text in the symbol table file. You can refer to the following command:

grep "T _text$"  output/buildroot/tmp_build/linux/System.map
ffff800008000000 T _text

With the information above, you can calculate the address offset using the following formula:

0x80200000 - 0xffff800008000000 = 0x800078200000

The offset 0x800078200000 is required for loading the symbol table. You can use the following command in Arm DS to load the symbol table for the initial boot phase in EL2:

add-symbol-file output/buildroot/tmp_build/linux/vmlinux  EL2N:0x800078200000

2. Load the debug symbol for the booting process at EL1 with MMU disabled and MMU enabled with ID mapping

After the Linux kernel transitions from EL2 to EL1, if the MMU is disabled or enabled with identity mapping, the symbol table offset remains the same as in the previous step, at 0x800078200000. Therefore, you can use the following command to load the symbol table:

add-symbol-file output/buildroot/tmp_build/linux/vmlinux  EL1N:0x800078200000

3. Load the debug symbol for EL1 with MMU enabled

After KASLR is disabled, the Linux kernel maps the physical address of the loaded Linux image to a virtual address, and this virtual address matches the address in the symbol table. Therefore, after the MMU is enabled, you can use the following command to add the symbol table:

add-symbol-file output/buildroot/tmp_build/linux/vmlinux  EL1N:0

4. Load the debug symbol for EL2 with MMU enabled

As mentioned in the previous step, the following command can be used to load the symbol table after the MMU is enabled in EL2:

add-symbol-file output/buildroot/tmp_build/linux/vmlinux  EL2N:0

In the Debugger tab of the Linux Kernel debug configuration, include corresponding commands in Execute Debugger Commands to add symbol tables, as shown in the following figure:

Each time the debugging connection starts, these commands are automatically executed to load the symbol table files

You can also add other commands, such as setting breakpoints:

break _text
break start_kernel

The commands will automatically execute after connecting to the target as follows:

How to start the Linux Kernel debugging

You can set any breakpoints at the Linux Kernel connection and click the Continue button as follows:

The Application Processor stops at the breakpoint and you can use the Continue button to continue debug the Linux Kernel as follows: