Porting Linux made easy with DS-5

Here at ARM, a colleague recently wanted to port Linux to a prototype of a new high-performance Cortex-A9 based platform.  To develop and debug this port, he needed to be able to set breakpoints, view registers, view memory, single-step at source level, and so on, in fact all the normal facilities provided by a debugger, but he wanted to do these both before the MMU is enabled (with a physical memory map), and after the MMU is enabled (with a virtual memory map).

The DS-5 Debugger has a slick Debug Configuration dialog in Eclipse that makes it easy to configure a debugging session to a target.  Predefined debug configuration types include "Bare Metal Debug", "Linux Application Debug", and "Linux Kernel and/or Device Driver Debug".  The latter is the topic of this blog.  This debug configuration type is primarily designed for post-MMU debug to provide full kernel awareness, but also has some extra features that allow it to be used for pre-MMU debug too.  This makes it possible to debug the Linux kernel, all the way from its entry point, through the pre-MMU stages, and then seamlessly through the MMU enable stage to post-MMU debug with full kernel awareness, all with source-level symbols and all without the need for tedious disconnecting/reconfiguring/ reconnecting!

Screenshot of Project type = "Linux Kernel and/or Device Driver Debug"

My colleague had already started using DS-5 Debugger's Debug Configuration Project type = "Linux Kernel and/or Device Driver Debug" to establish a connection to his target but was seeing data aborts occurring when a breakpoint was being hit at the entry point, after loading the kernel's debug symbols from the vmlinux file.  This was because vmlinux contains virtual addresses so usage of it assumes the OS is up and running with the MMU enabled.  After loading the debug symbols from vmlinux with the "file" command, the DS-5 Debugger will attempt to access various symbol locations in the kernel to support its kernel awareness, for example, "init_nsproxy.uts_ns->name" to get the kernel name and version, to support its "info os-version" command.  If the MMU is not yet on, it will access invalid addresses, leading to data aborts occurring.  However, this kernel awareness support feature can be temporarily disabled during the pre-MMU stage with the CLI command "set os enabled off", and later (post-MMU) re-enabled with the CLI command "set os enabled on".  Furthermore, the debug symbols in vmlinux are virtual addresses and can't be used as-is until the MMU is on, but can be used to debug pre-MMU at source-level if an offset is applied to them.

Let's consider the case where the kernel is loaded and started by a bootloader such as U-Boot.

The general sequence is:

• The bootloader initializes the platform, for example, sets-up the memory controllers
• The bootloader reads the kernel from FLASH or SD card into RAM at some physical address
• The bootloader prepares the kernel parameters (known as ATAGs) including the kernel command line (bootargs), memory description and machine ID, and puts the information into physical RAM and registers $r1 and $r2 ($r0 == 0)
• The bootloader jumps to the loaded kernel image at offset zero.
  At this point the MMU is off and all addresses are physical.  The symbols in the kernel debug info (in vmlinux) are virtual addresses and can't be used as-is until the MMU is on.  But below, we'll show how to use them with an offset.
• The kernel is typically compressed (where uImage is a compressed zImage with a U-Boot header).  A compressed kernel starts by decompressing itself and then jumps to its entry point, for example, 0x80008000.  Older kernels get decompressed into some higher memory, then get moved back to the entry point in lower memory.  More recent kernels more efficiently get moved to higher memory first, then get decompressed into lower memory.  For an uncompressed kernel (where uImage contains an uncompressed Image with a U-Boot header), this step does not apply.
• The uncompressed kernel starts with the MMU off and uses position-independent code to examine the parameters passed by the bootloader.  It sets up the MMU tables and then in the function __turn_mmu_on it turns the MMU on and jumps to the virtual address __mmap_switched.

Now we will look at how the above sequence would work with a specific example using DS-5.

To debug the kernel at source-level, the kernel must be built with debug info: CONFIG_KERNEL_DEBUG=y, CONFIG_DEBUG_INFO=y, and (in kernel versions 2.6.36 and later) CONFIG_DEBUG_INFO_REDUCED=n.

For DS-5 users, a ready-made example Linux distribution for BeagleBoard is provided, already built with debug info, complete with SD-Card image, vmlinux symbols file, file system, and full source code.  To debug the kernel at source level, unzip/untar source.tar.bz2.  Program a blank SD-Card with the supplied SD-Card image as described in the DS-5 documentation, then insert the SD-Card into the BeagleBoard.

• Boot the target and stop it in the bootloader.
  For DS-5 Example Linux Distribution on BeagleBoard: With the pre-programmed SD-Card inserted, power-up/reset BeagleBoard, and interrupt U-Boot's count-down sequence by typing any character to the serial console.
• Connect DS-5 Debugger to the target, using the Run->Debug Configuration:
  Project type = "Linux Kernel and/or Device Driver Debug"
  Target connection = "DSTREAM"
  Debugger Run Control = "Connect only"
• Stop the target (press the Suspend button) and set a hardware breakpoint on the physical address where the bootloader is going to load/jump to the kernel.  You'll have to consult the bootloader to find out the address.
  For DS-5 Example Linux Distribution on BeagleBoard, this will be:
  hbreak *0x80008000
  
  
• Restart the target with the Run/Continue button, or press F8.
• Make the bootloader load/boot the kernel
  For DS-5 Example Linux Distribution on BeagleBoard, type at the U-Boot prompt:
  run bootcmd
  or just:
  boot
  
  
• A compressed kernel will stop at the breakpoint at 0x80008000 before decompressing.  Run/Continue the target (press F8) again.  After decompressing, code execution will stop at 0x80008000 again, at the kernel entry point.  The Disassembly view shows the assembly code at symbol "stext" in /source/arch/arm/kernel/head.S, but no source code is shown yet, because the vmlinux symbols have not yet been loaded - this is done later.
• At the kernel entry point, you can check the CPU and CP15 registers:
  - the CPU is in SVC (supervisor) mode with both IRQ and FIQ interrupts disabled
  - R0 is 0
  - R1 contains the ARM Linux machine type, for example, 0x060A = 1546 = BeagleBoard.  See: http://www.arm.linux...loper/machines/
  - R2 contains a pointer to all the kernel parameters (ATAGs)
  - the MMU is off
  - the Data cache is off
  - the Instruction cache may be either on or off
• As the MMU is currently off, DS-5 Debugger's OS support must be disabled before loading debug symbols to avoid data aborts occurring, with:
  set os enabled off
• For source-level debug before the MMU is on you need to calculate the offset between the virtual and physical addresses of the code.  For instance if the kernel is linked at virtual address 0xC0008000 and has been loaded at physical address 0x80008000 (as is the case with the DS-5 Example Linux Distribution), the offset is -0x40000000 (which is 0x80008000 - 0xC0008000).
  To determine the virtual address at which the kernel is linked, use:
  readelf -h vmlinux
  or
  fromelf vmlinux
  and look for the image entry point address entry in the ELF file header.
  
  Now load the kernel debug symbols, with the offset:
  file
  add-symbol-file vmlinux -0x40000000
  
  

• Right-mouse-click in the Disassembly view, and select "Show in Source".  DS-5 Debugger will try to open head.S in its Editor view.  If you are using the ready-made vmlinux file provided with DS-5, that was built using different directory paths than your host machine, you will need to set a source path substitution to point the debugger at the kernel sources in .../distribution/kernel/linux-2.6.35-patched/source/, from the untar'd source.tar.bz2.  Make sure that both the "Image Path" and "Host Path" both end with a corresponding directory; for example,  ".../foo/linux-2.6.35-patched/source/" and "...\bar\linux-2.6.35-patched\source\".  Use of a shorter path is usually best.  Then right-mouse-click again in the Disassembly view, and select "Show in Source".  head.S will open in the Editor view.

  You can now set breakpoints, view registers, view memory, single-step,  and other usual debug operations at this pre-MMU stage, all at source  level.

  Screenshot of pre-MMU source-level debug

• To see when the MMU will be turned on, set a breakpoint:
  tbreak __turn_mmu_on
  Run/Continue the target (press F8).
  When __turn_mmu_on is reached, note the value of R13 - this contains the virtual address of __mmap_switched and is the place the code will jump to after the MMU is enabled.
• It is not possible to single-step through __turn_mmu_on, so place a hardware breakpoint on the virtual address of __mmap_switched:
  thbreak *$r13
  then Run/Continue the target (press F8). Now you will reach the breakpoint and the MMU is on.
  
  If you don't need to debug before the MMU is on and you know the virtual address of __mmap_switched either because you saw it before in $r13 or you got it from the vmlinux file, for example, using nm to list symbols from object files:
  nm .../vmlinux | grep __mmap_switched
  then you can skip steps 3 to 12, even if your bootloader is loading a compressed kernel, by using:
  thbreak *[address of __mmap_switched]
  then running to it.
• Ensure OS support is off, then discard the symbols and reload the symbols without an offset with:
  set os enabled off
  file
  add-symbol-file vmlinux
  
  You can now set breakpoints, view registers, view memory, single-step, and other usual debug operations at this post-MMU stage, all at source level.
• Before OS support can be enabled in the debugger, set a breakpoint on start_kernel() (in \source\init\main.c, the main C code entry into the kernel after all the architecture-specific setup has been done), with:
  tbreak start_kernel
  then run to it.  You can now enable OS support with:
  set os enabled on
• To confirm the kernel symbols match the kernel image, try reading the Linux version information with:
  info os-version
  
  which will show e.g:
  Operating system version: Linux 2.6.38.7 #1 PREEMPT Tue May 8 15:45:31 BST 2012 arm
  
  This is equivalent to:
  print init_nsproxy.uts_ns->name
  
  which will show e.g:
  $1 = {sysname = "Linux", nodename = "", release = "2.6.38.7", version = "#1 PREEMPT Tue May 8 15:45:31 BST 2012", machine = "arm", domainname = "(none)"}
  
  
• Another very useful feature during kernel bring-up is to display early printk output in the DS-5 command window, in particular, before the console has been enabled, so there would be no output from the serial port.
  
  For example, set a breakpoint on console_init() in main.c, and run to it.  No log messages have yet been output to the console because it hasn't yet been enabled, but the entire log so far can still be viewed with:
  info os-log
  
  To view the log output line by line, as it happens, use:
  set os log-capture on

Screenshot of post-MMU source-level debug

Many of the above steps can be automated, either with a script file, or by filling-in the Debug Configuration's fields before launching.

In summary, we have looked at how DS-5 can be used to debug the Linux kernel, both pre-MMU enable and post-MMU enable stages, to help porting Linux to a new platform.