Arm Development Studio Platinum 2021.a is required to debug RD_N2 FVP. 

Download Software Stack

To build edk2, we will use the Reference Design (RD) software stack. This contains lots of helpful scripts for building and running UEFI on Arm FVPs.

Select a RD platform. There is a list of platforms here. For this guide we are using RD-N2 release tag RD-INFRA-2021.09.17. Previous versions will not work.

For this guide, the software stack has been installed to ~/rd-infra.

Download the software stack using the guide here.

Setup Software Stack for Debugging

Make sure model-scripts/sgi/uefi.sh contains platforms_rdinfra[rdn2]=1

~/rd-infra$ grep "platforms_rdinfra\[rdn2\]" model-scripts/sgi/uefi.sh
platforms_rdinfra[rdn2]=1

Modify EDK2 to prevent build errors

The ArmCacheMaintenanceLib module causes the build to fail when space optimisation is disabled. Without optimisation, the module uses the Global Offset Table which is not supported by the EDK II tools. To fix this, we will re-enable space optimisation for the ArmCacheMaintenanceLib module only. This makes debugging this specific module more difficult, however it's quite a small module so it's still fairly easy to figure out what's going on.

In uefi/edk2/ArmPkg/Library/ArmCacheMaintenanceLib/ArmCacheMaintenanceLib.inf, Under the [Defines] section, add a new section [BuildOptions]:

[BuildOptions]
  *_*_*_CC_FLAGS  = -Os

Modify tools_def 

In uefi/edk2/BaseTool/Conf/tools_def.template remove -Os from line DEFINE GCC_ALL_CC_FLAGS. This makes debugging much easier by not space optimising the firmware. 

uefi/edk2/BaseTool/Conf/tools_def.template: 

DEFINE GCC_ALL_CC_FLAGS         = -g -fshort-wchar -fno-builtin -fno-strict-aliasing -Wall -Werror -Wno-array-bounds -include AutoGen.h -fno-common

Add –g to DEFINE GCC_ASM_FLAGS to generate debug information. 

DEFINE GCC_ASM_FLAGS              = -g -c -x assembler -imacros AutoGen.h

Update uefi/edk2/Conf/tools_def.txt if you already built the stack previously. 

~/rd-infra$ cp uefi/edk2/BaseTools/Conf/tools_def.template uefi/edk2/Conf/tools_def.txt

Set DebugPrintErrorLevel 

In uefi/edk2/edk2-platforms/Platform/ARM/SgiPkg/RdN2/RdN2.dsc add the following line in section PcdsFixedAtBuild.common to enable verbose printing. 

gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0x00000004

Set 0x00000004 to a combination of these flags: 

#define DEBUG_INIT      0x00000001  // Initialization
#define DEBUG_WARN      0x00000002  // Warnings
#define DEBUG_LOAD      0x00000004  // Load events
#define DEBUG_FS        0x00000008  // EFI File system
#define DEBUG_POOL      0x00000010  // Alloc & Free (pool)
#define DEBUG_PAGE      0x00000020  // Alloc & Free (page)
#define DEBUG_INFO      0x00000040  // Informational debug messages
#define DEBUG_DISPATCH  0x00000080  // PEI/DXE/SMM Dispatchers
#define DEBUG_VARIABLE  0x00000100  // Variable
#define DEBUG_BM        0x00000400  // Boot Manager
#define DEBUG_BLKIO     0x00001000  // BlkIo Driver
#define DEBUG_NET       0x00004000  // Network Io Driver
#define DEBUG_UNDI      0x00010000  // UNDI Driver
#define DEBUG_LOADFILE  0x00020000  // LoadFile
#define DEBUG_EVENT     0x00080000  // Event messages
#define DEBUG_GCD       0x00100000  // Global Coherency Database changes
#define DEBUG_CACHE     0x00200000  // Memory range cachability changes
#define DEBUG_VERBOSE   0x00400000  // Detailed debug messages that may
                                    // significantly impact boot performance
#define DEBUG_ERROR     0x80000000  // Error

from uefi/edk2/MdePkg/Include/Library/DebugLib.h 

DEBUG_LOAD must be enabled to see where dynamically loaded modules are.

Install FVP 

Download the correct FVP version for the platform and version you selected at the link here. 

For RD-N2 version RD-INFRA-2021.05.27 navigate to Neoverse N2 Reference Design FVP and click Download RD-N2. 

Run the installer FVP_RD_N2.sh. 

Modify FVP start script 

Modify model-scripts/rdinfra/platforms/rdn2/run_model.sh: 

remove -R parameter from PARAMS= section. 

The -R parameter tells the model to run as soon as the debug server has started, without waiting for a connection.

When we remove it, the model will wait for a debugger connection before starting.

PARAMS="--data css.scp.armcortexm7ct=$OUTDIR/scp_ramfw.bin@0x0BD80000 \
	--data css.mcp.armcortexm7ct=$OUTDIR/mcp_ramfw.bin@0x0BF80000 \
	-C css.mcp.ROMloader.fname=$OUTDIR/mcp_romfw.bin \
	-C css.scp.ROMloader.fname=$OUTDIR/scp_romfw.bin \
	-C css.trustedBootROMloader.fname=$OUTDIR/$BL1_IMAGE \
	-C board.flashloader0.fname=$OUTDIR/$FIP_IMAGE \
	-C board.flashloader1.fname=$PWD/nor1_flash.img \
	-C board.flashloader1.fnameWrite=$PWD/nor1_flash.img \
	-C board.flashloader2.fname=$PWD/nor2_flash.img \
	-C board.flashloader2.fnameWrite=$PWD/nor2_flash.img \
	-S \
	-C css.scp.pl011_uart_scp.out_file=${MODEL_TYPE,,}/${UART0_SCP_OUTPUT_FILE_NAME} \
	-C css.scp.pl011_uart_scp.unbuffered_output=1 \
	-C css.scp.pl011_uart_scp.uart_enable=true \

Build EDK2

~/rd-infra$ ./build-scripts/build-test-uefi.sh -p rdn2 all

Debug RDN2 

Find the start address of UEFI module 

TF-A loads BL33_AP_UEFI binary (the UEFI firmware) to the address specified by PLAT_ARM_NS_IMAGE_BASE in arm-tf/include/plat/arm/css/common/css_dev.h 

~/rd-infra$ grep PLAT_ARM_NS_IMAGE_BASE arm-tf/include/plat/arm/css/common/css_def.h

#define PLAT_ARM_NS_IMAGE_BASE U(0xE0000000)

The start address is 0xE0000000 

Start FVP 

Update MODEL environment variable with your FVP_RD_N2 model executable. 

export MODEL=~/FVP_RD_N2/models/Linux64_GCC-6.4/FVP_RD_N2

Go to model-scripts/rdinfra and run uefi.sh -p rdn2 

~/rd-infra$ cd model-scripts/rdinfra/ 

~/rd-infra/model-scripts/rdinfra$ ./uefi.sh -p rdn2 

The FVP will now start and wait for a debugger to connect. 

Create a debug connection 

Launch ArmDS Platinum 2021.a 

Create a new debug connection 

Select "Model Connection" and next

Enter a name for the connection.

Select "Add a new Model"

Select CADI as the model interface

Select "Browse for model running on local host"

Once your model terminal says “CADI Debug Server started for ARM Models...” the model should appear in arm DS.

Click Finish. The edit configuration window should then appear. 

Under “Connection” select Imported/Infra5_RD_N2/Bare Metal Debug/ARM_Neoverse-N2x15 Multi-Cluster SMP

Under Debugger, check “Execute Debugger Commands” and enter a breakpoint for the UEFI entry point we found earlier. 

Click Debug. The debugger should connect to the FVP. 

Load Debug Symbols 

If you have not connected to the FVP, click "Connect" to connect.

Currently it has stopped at EL3:0x00000000. This is the TF-A firmware. Click continue (F8)  to start execution. It will then break at EL2N:0xE0000000. 

Hit Continue (F8) again to let the FVP load into UEFI uninterrupted. This will load all the modules you would like to debug so that the debugger knows where they are.

Do the same actions in the same order you will when you are debugging as this affects the locations of the modules. For example, open UEFI interface -> Enter UEFI Shell -> Load EFI driver in the same order each time.

Take note of the file name of the UART log. This can be found in the terminal window where you ran uefi.sh. The UART log contains the addresses of the module locations and will be used to load the debug symbols.

Once you have loaded into the UEFI menu, stop the FVP by pressing Ctrl+C in the terminal window where you started the model, then press disconnect in the Arm DS window.

In Arm DS, go to the scripts tab and hit "Import a script or directory" then "Import a DS or Jython script"

Select the script "~/rd-infra/uefi/edk2/ArmPlatformPkg/Scripts/Ds5/cmd_load_symbols.py".

Right click the script in the list and select "Parameters"

Enter the following parameters

-f (<UEFI entry point>, 0x20000) The UEFI entry point was found earlier (0xE0000000). 0x20000 is the size which can be found in the UEFI build log.

-a -v

-i <path to UART log file> This is the path to the UART log which was produced by a running the whole UEFI process

-o <path to objdump executable> This is the path to aarch64-none-linux-gnu-objdump. If you followed the installation instructions, this will be ~/rd-infra/tools/gcc/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-objdump

The script has now been setup.

Start the FVP again, hit the connect button in Arm DS and press continue into the UEFI firmware (0xE0000000). Load the debug symbols by right clicking the script and selecting "Run".

All the debug symbols have loaded, you can now start debugging!

Static vs Dynamic Modules

If you are debugging SEC or PEI you need to be aware of whether your module has a dynamic address.

Before PEI is reloaded, all the module addresses are defined at build time, however once the MMU has been initialised, modules are loaded and allocated memory. The dynamic addresses are the same between runs, however they may be different if you rebuild EDK2, or if you perform actions in a different order. If you are debugging a module before PeiCore has been reloaded, remove the -i and -o arguments from the cmd_load_symbols script so that the dynamic modules aren't loaded. The debug symbols for some modules will be defined twice if you don't do this, which could cause some symbols to not be loaded.

UEFI Boot Process

Now that the debug symbols are loaded, we can run through the UEFI boot process.

For more details on the boot process, check out the UEFI and PEI specifications.

Security (SEC)

First we enter the Security (SEC) stage. This is located where the branch instruction at the UEFI entry point (0xE0000000) is pointing. (0xE0017FD8).

SEC does some setup and passes control to PEI.

The module is located at uefi/edk2/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf. It is called ArmPlatformPrePeiCore.

Pre-EFI Initialization (PEI)

PEI sets up hardware abstractions of devices like Firmware and Memory for DXE to use. It does this searching the Firmware Volumes for PEIMs, loads them, then attempts to dispatch them.

When MemoryPeim is dispatched, which sets up the MMU, the PeiCore is reloaded into memory at a dynamic memory address.

PEI Foundation (PeiCore) entry point is located in uefi/edk2/MdePkg/Library/PeiCoreEntryPoint/PeiCoreEntryPoint.c

Pre-EFI Initialization Modules (PEIMs)

Pre-EFI Initialization Modules create an abstraction of a specific piece of hardware. They can work with other PEIMs using PPI (PEIM to PEIM Interface).

The PEI dispatcher calls one PEIM at a time, invoking it's entrypoint function. These functions perform initialization for the PEIM, which then register a service. This could be a PPI service for other PEIMs to call, or a HOB (Hand Off Block) which can be passed to DXE.

Now that all PEIMs are loaded and all corresponding HOBs have been created, PeiCore uses Dxe Initial Program Load (DxeIpl) with a list of HOBs to call DxeCore.

Drive Execution Environment (DXE)

The entrypoint of the DxeCore is DxeMain()

Similar to PEI, DXE traverses the Firmware Volumes looking for DXE drivers. It loads each driver and transfers control to them. DXE drivers can perform initialization, which could involve installing protocols which later stages can use. These drivers could be to enable input or output, or could be drivers for boot devices, like SD card or Network.

DxeMain initialises exception handlers, debug agent for DXE phase, DXE memory subsystem, EFI system table, EFI Runtime Services Table, Image services and Event services.

One of the drivers loaded is BdsDxe. This installs gEfiBdsArchProtocolGuid, which is used to jump to the BDS entry and transitions to the BDS stage.

Boot Device Selection (BDS)

BDSDxe reads the boot order list from EFI variables.

Boot timeout EFI is evaluated and user input from keyboard and mouse is connected. UI applications are shown.

Debugging Runtime Modules

Once the Operating System has been loaded, the OS is given access to UEFI Runtime Services. These services are loaded during the DXE stage, however their addresses are remapped from the Physical Memory to the OS’s virtual memory so we need to reload the debug symbols to debug them. Unfortunately, these addresses are not logged anywhere so the process to find them is slightly more involved than normal UEFI modules. 

Load Debug Symbols for UEFI

Build UEFI and Busybox and start the FVP. Instead of building just UEFI, build busybox as well. Instead of running build-test-uefi.sh, run build-scripts/rdinfra/build-test-busybox.sh, and instead of running uefi.sh to start, run boot.sh.

Load the debug symbols as shown above.

Identify the physical addresses of the modules to debug

Find which module you'd like to debug. For this guide we will debug RealTimeClock located in uefi/edk2/EmbeddedPkg/RealTimeClockRuntimeDxe.

Head to the Command tab of Arm DS. There will be a list of add-symbol-file commands which the script called to load the debug symbols. Look for the command corresponding to the module you’d like to debug. There will be a path to a .dll file and an address. Note both down. 

Set a breakpoint where the addresses are mapped

We will now set a breakpoint at the point where the runtime modules are remapped into virtual address space.  

In Arm DS, press File > Open file and open uefi/edk2/MdeModulePkg/Core/RuntimeDxe/Runtime.c 

Set a breakpoint on line 317, where PeCoffLoaderRelocateImageForRuntime is called.

Press “Continue” on the debugger. In the uart console, wait for the “Press ESCAPE for boot options” to time out. GRUB will then load. Press enter on “RD-N2 BusyBox” 

When the debugger breaks, go to the variables tab and expand “RuntimeImage” to look at “ImageBase”. 

The firmware is currently going through each runtime module and translating it’s address. 

Press continue on the debugger until ImageBase matches the entrypoint we found earlier. (0xF7F10000) 

Now that ImageBase matches, look at VirtImageBase. This is the Virtual Address of the module. 

Reload debug symbols at the new address

Go to the command tab and enter

add-symbol-file <path to module dll> <module virtual address>

Hit submit and the debug symbols for the module are now loaded at the virtual address!

We can now set a breakpoint in RealTimeClock.c to break when the OS calls the Runtime Service. 

Hit file > Open File and open uefi/edk2/EmbeddedPkg/RealTimeClockRuntimeDxe/RealTimeClock.c

One of the functions available is GetTime. Set a breakpoint there.

Hit continue until the new breakpoint is reached. It takes a while for the linux kernel to load, however we have now hit the breakpoint while the kernel is loading!