To develop, port and debug the Linux kernel on a platform, you will need to be able to set breakpoints, view registers, view memory, single-step at source level and so on - all the normal facilities provided by a debugger. You may also need to do these both before the MMU is enabled (with a physical memory map), and after the MMU is enabled (with a virtual memory map). DS-5 Debugger in Arm DS-5 Development Studio allows you to do all this and more, not just for single-core platforms, but for multi-core SMP platforms too.
This blog shows how to debug the Linux 4.12.0 kernel from the Linaro 17.10 distribution, running on a Armv8-A Fixed Virtual Platform (FVP) model – the AEMv8Ax4 (FVP_Base_AEMv8A). The model is supplied with DS-5 Ultimate Edition. The Armv8-A Linux kernel, pre-built for debug, complete with vmlinux symbol file, file system, and full source code, are required for this tutorial. The boot loader used here is the Trusted Firmware and U-Boot.
Screen-shot of post-MMU source-level debug
DS-5 Debugger has a slick Debug Configuration dialog in Eclipse that makes it easy to configure a debugging session to a target. The Linux Kernel Debug configuration type is primarily designed for post-MMU debug to provide full kernel awareness but, with some extra controls, can also be used for pre-MMU debug too. This makes it possible to debug the Linux kernel, from its entry point, through the pre-MMU stages, and then seamlessly through the MMU enable stage to post-MMU debug with full kernel awareness, all the way to the login prompt, all with source-level symbols, and all without the need for tedious disconnecting, reconfiguring and reconnecting.
FVP models make use of "telnet" as a serial terminal, to transfer serial data from application code running on the FVP model via a modelled UART. The telnet client is not enabled by default on Windows 7 and later. If the telnet client is disabled on your computer, you may see the error: “Windows cannot find C:\Windows\System32\telnet.exe” or “No telnet executable was found on your system”.
To enable the telnet client:
This section describes how to obtain the pre-built Linaro Linux distribution and launch the AEMv8A model. This does not need to be done if you intend to debug Linux on real hardware that has Linux already installed.
FVP_Base_AEMv8A -C bp.secure_memory=0 -C cache_state_modelled=0 -C bp.pl011_uart0.untimed_fifos=1 -C bp.secureflashloader.fname=bl1.bin -C bp.flashloader0.fname=fip.bin --data cluster0.cpu0=Image@0x80080000 --data cluster0.cpu0=fvp-base-aemv8a-aemv8a.dtb@0x82000000 --data cluster0.cpu0=ramdisk.img@0x84000000 -C bp.ve_sysregs.mmbSiteDefault=0 -C bp.virtioblockdevice.image_path=..\lt-vexpress64-openembedded_minimal-armv8-gcc-4.9_20150912-729.img -C bp.smsc_91c111.enabled=true -C bp.hostbridge.userNetworking=true -C bp.hostbridge.userNetPorts="5555=5555,8080=8080,22=22" --cadi-server
The --cadi-server parameter tells the model to wait for a debugger to be connected.
To capture instruction execution history ("trace") too, add, e.g.:
--plugin="C:\Program Files\DS-5 v5.28.0\sw\models\bin\MTS.dll" (Windows) or --plugin=".../DS-5 v5.28.0/sw/models/bin/MTS.so" (Linux)
--plugin="C:\Program Files\DS-5 v5.28.0\sw\models\bin\MTS.dll"
The following sequence is for using DS-5 on a Windows host, but Linux is the same except for the pathnames.
Debug Configuration for Linux Kernel Debug on AEMv8Ax4 FVP
Example of the target being stopped at the entry point of the Trusted Firmware
Example of the breakpoint at the entry point of the kernel
Code execution is stopped at the breakpoint, and the Disassembly view shows the assembly code, but no source code is shown yet, because the vmlinux symbols have not yet been loaded. In the Command entry field (at the bottom of the Commands view) enter: set os physical-address 0x80080000add-symbol-file "path\to\vmlinux"
Fix-up the “\path\to” to point to the real folder on your hard-disk where the vmlinux file can be found, for example, inside fvp-latest-oe-debug.
Explanation: Debug symbols in the vmlinux file have virtual addresses, so can normally only be used to debug when Linux is up and running with the MMU enabled. To debug pre-MMU physical addresses at source-level, an offset must be applied to the addresses in the vmlinux file. The offset between the physical and virtual addresses is calculated by DS-5 Debugger using the physical address given in the set os physical-address command and the virtual address read from the ELF header of the vmlinux file. The physical address at which the kernel is loaded comes from the model parameter: --data cluster0.cpu0=Image@0x80080000, which matches the U-Boot environment variable kernel_addr. You may need to change the address given with the set os physical-address command to match your own kernel.
Check the following appears in the Command view, to confirm Linux kernel support has been enabled:Enabled Linux kernel support for version "Linux 4.12.0 #1 SMP PREEMPT Thu Nov 16 07:37:36 UTC 2017 aarch64"
Symbol names now appear in the Disassembly view, with the PC at the symbol stext.
DS-5 Debugger will try to open .../arch/arm64/kernel/head.S in its Editor view. If it does not find the kernel sources in paths relative to those specified in the vmlinux file, a button appears inviting you to set a path substitution:
Press on the button to open a dialog to set a substitute source path, to re-direct paths from where the kernel was built to where the kernel source tree is located on your hard-disk.
If you need to revisit these paths later, this dialog can be opened by clicking on View Menu (the upside-down triangle) on the Debug Control view toolbar, then selecting Path Substitution…
That’s all the set-up now done. You can now view registers, view variables, view memory, set breakpoints and watchpoints, single-step, and all the other usual debug operations at this pre-MMU stage, all at source level.
Screen-shot of pre-MMU source-level debug
In the Registers view, expand AArch64, then expand the Core and System register groups. At the kernel entry point, you can check the Core and System registers are set as recommended by: Booting AArch64 Linux post on the Kernel.org site
The kernel code here executes in EL2, and the kernel symbols have been loaded for EL2, but most of the kernel code later runs in EL1N. To see where the change from EL2 to EL1N occurs, search head.S using Edit > Find/Replace… for the ‘eret’ instruction. It is at the end of ‘el2_setup’:
: mov w0, #BOOT_CPU_MODE_EL2 // This CPU booted in EL2 eretENDPROC(el2_setup)
mov w0, #BOOT_CPU_MODE_EL2 // This CPU booted in EL2 eretENDPROC(el2_setup)
Double-click on the far left-hand side of the ‘eret’ line to set a breakpoint, then Continue (or press F8) running to it. The processor is still currently in EL2: Execution stopped in EL2h mode at breakpoint 1: EL2:0x00000000808BC138On core ARMAEMv8-A_MP_0 (ID 0)EL2:0x00000000808BC138 527,0 eret Now single Step Source Line (or press F5). The processor switches to EL1N: Execution stopped in EL1h mode at EL1N:0x0000000080B00008On core ARMAEMv8-A_MP_0 (ID 0)In head.SEL1N:0x0000000080B00008 119,0 adrp x23, __PHYS_OFFSET
DS-5 Debugger automatically reloads the vmlinux symbols, and applies them to the EL1N pre-MMU memory space. The Editor view updates automatically using these new symbols to show the correct position of the PC in the source file head.S.
To see where the MMU will be turned on, set a breakpoint with: thbreak __enable_mmuthen Continue running (or press F8).Note: in earlier kernels, this function was named __turn_mmu_on. In some even earlier kernels, it had file-static scope, so a special notation was needed:thbreak EL1N:("head.o"::__turn_mmu_on)
The instruction trace shows the history of instruction execution to reach this point – feel free to explore this view here.
When __enable_mmu is reached, the value in register x27 will contain the virtual address of __mmap_switched, where the code will jump to after the MMU is enabled.
You can now view registers, view variables, view memory, set breakpoints and watchpoints, single-step, and all the other usual debug operations at this post-MMU stage, all at source level.
So far, CPU 0 has been doing all the work – the other (“secondary”) CPUs are still “asleep” (in WFI) awaiting an interrupt. You can check this by selecting one of the other cores in the Debug Control view. The next steps explore the release of the secondary processors.
Secondary processors are released from their holding pen by CPU0 executing smp_prepare_cpus(). Set a breakpoint on this with: thbreak smp_prepare_cpusthen Continue (F8) running to it.In the Debug Control view, notice that Core 0 is in C code with its MMU on, and has started some threads, but the other cores are still “asleep” awaiting an interrupt (WFI) with their MMU off. Set a breakpoint (note: not a temporary one this time) with:hbreak secondary_start_kernelthen Continue (F8) running to it.In Debug Control view, notice that Core 1 has woken up and has stopped at the breakpoint.Run again. Core 2 will hit the same breakpoint. Run again. Core 3 will hit the same breakpoint. All secondary CPUs have now been released from the holding pen. The secondary CPUs all then enter cpu_do_idle() until the primary CPU gives them some work to do.
Many of the above steps can be automated, either with a script file, or by filling-in the Debug Configuration's fields before launching.
Now that the kernel is up and running, you can debug with full thread awareness.
The Debug Control view is currently showing the cores, but can be changed to show the threads. In the Debug Control view, either click on the Display Threads button in the toolbar, or right-mouse-click on the connection, then select Display Threads. The current thread (“swapper”), “Active Threads” and “All Threads” appears in the Debug Control view.
Note the context for the (linked) Trace view changes from “core” to “thread” too, so that you can now see in the Trace view the instructions that were executed for the current thread.
A useful feature during kernel bring-up is to be able to view earlyprintk (also known as bootconsole) and regular Console output, in particular, if the console is not enabled (so there would be no output from the serial port), or if you have no serial port connected. The entire log so far can be viewed with: info os-log
Screen-shot of log capture in progress
To view the log output line by line, as it happens, use: set os log-capture on
In summary, we have looked at how DS-5 can be used to debug the Armv8-A Linux kernel, at both pre-MMU enable and post-MMU enable stages, and looked at a few of the kernel’s internal features.
Please see the DS-5 tutorials for other approaches for debugging and optimization of your Arm-based projects.