Arm Development Studio Platinum 2021.a is required to debug RD_N2 FVP. 

Download Software Stack

To build edk2, we will use the Reference Design (RD) software stack. This contains lots of helpful scripts for building and running UEFI on Arm FVPs.

Select a RD platform. There is a list of platforms here. For this guide we are using RD-N2 release tag RD-INFRA-2021.05.27.

For this guide, the software stack has been installed to ~/rd-infra.

Download the software stack using the guide here.

Setup Software Stack for Debugging

Make sure model-scripts/sgi/uefi.sh contains platforms_rdinfra[rdn2]=1

~/rd-infra$ grep "platforms_rdinfra\[rdn2\]" model-scripts/sgi/uefi.sh
platforms_rdinfra[rdn2]=1

Build UEFI edk2 

Now build the software stack. This generates the config files we need to modify. 

~/rd-infra$ ./build-scripts/build-test-uefi.sh -p rdn2 all

Modify tools_def 

In uefi/edk2/Conf/tools_def.txt and uefi/edk2/BaseTool/Conf/tools_def.template remove -Os from line DEFINE GCC_ALL_CC_FLAGS. This makes debugging much easier by not space optimising the firmware. 

uefi/edk2/Conf/tools_def.txt: 

DEFINE GCC_ALL_CC_FLAGS         = -g -fshort-wchar -fno-builtin -fno-strict-aliasing -Wall -Werror -Wno-array-bounds -include AutoGen.h -fno-common

Add –g to DEFINE GCC_ASM_FLAGS to generate debug information. 

DEFINE GCC_ASM_FLAGS              = -g -c -x assembler -imacros AutoGen.h

Optional: Update tools_def.template to reflect these changes in the future. 

~/rd-infra$ cp uefi/edk2/Conf/tools_def.txt uefi/edk2/BaseTools/Conf/tools_def.template

Fix build errors

In uefi/edk2/edk2-platforms/Platform/ARM/SgiPkg/Drivers/PlatformDxe/PlatformDxe.inf under section Protocols add gHestTableProtocolGuid.

[Protocols] 
  gEfiAcpiTableProtocolGuid          ## CONSUMES 
  gEfiAcpiSdtProtocolGuid            ## CONSUMES 
  gHestTableProtocolGuid             ## SOMETIMES CONSUMES 

When space optimisation is enabled, the code which locates the HEST table is automatically removed. However, as we will disable space optimisation, the HEST table code will be there but the linker will see that it's not referenced anywhere, causing the build to fail. We add gHestTableProtocolGuid so that it is referenced somewhere in the code.

Also having the Secure Partition Monitor (SPM) enabled causes the build to fail when space optimisation is disabled, so we will disable SPM.

In build-scripts/configs/rdn2/rdn2 make ARM_TF_ENABLE_SPM=0

Set DebugPrintErrorLevel 

In uefi/edk2/edk2-platforms/Platform/ARM/SgiPkg/RdN2/RdN2.dsc add the following line in section PcdsFixedAtBuild.common to enable verbose printing. 

gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0x00000004

Set 0x00000004 to a combination of these flags: 

#define DEBUG_INIT      0x00000001  // Initialization
#define DEBUG_WARN      0x00000002  // Warnings
#define DEBUG_LOAD      0x00000004  // Load events
#define DEBUG_FS        0x00000008  // EFI File system
#define DEBUG_POOL      0x00000010  // Alloc & Free (pool)
#define DEBUG_PAGE      0x00000020  // Alloc & Free (page)
#define DEBUG_INFO      0x00000040  // Informational debug messages
#define DEBUG_DISPATCH  0x00000080  // PEI/DXE/SMM Dispatchers
#define DEBUG_VARIABLE  0x00000100  // Variable
#define DEBUG_BM        0x00000400  // Boot Manager
#define DEBUG_BLKIO     0x00001000  // BlkIo Driver
#define DEBUG_NET       0x00004000  // Network Io Driver
#define DEBUG_UNDI      0x00010000  // UNDI Driver
#define DEBUG_LOADFILE  0x00020000  // LoadFile
#define DEBUG_EVENT     0x00080000  // Event messages
#define DEBUG_GCD       0x00100000  // Global Coherency Database changes
#define DEBUG_CACHE     0x00200000  // Memory range cachability changes
#define DEBUG_VERBOSE   0x00400000  // Detailed debug messages that may
                                    // significantly impact boot performance
#define DEBUG_ERROR     0x80000000  // Error

from uefi/edk2/MdePkg/Include/Library/DebugLib.h 

DEBUG_LOAD must be enabled to see where dynamically loaded modules are.

Install FVP 

Download the correct FVP version for the platform and version you selected at the link here. 

For RD-N2 version RD-INFRA-2021.05.27 navigate to Neoverse N2 Reference Design FVP and click Download RD-N2. 

Run the installer FVP_RD_N2.sh. 

Modify FVP start script 

Modify model-scripts/rdinfra/platforms/rdn2/run_model.sh: 

remove -R parameter from PARAMS= section. 

The -R parameter tells the model to run as soon as the debug server has started, without waiting for a connection.

When we remove it, the model will wait for a debugger connection before starting.

PARAMS="--data css.scp.armcortexm7ct=$OUTDIR/scp_ramfw.bin@0x0BD80000 \
	--data css.mcp.armcortexm7ct=$OUTDIR/mcp_ramfw.bin@0x0BF80000 \
	-C css.mcp.ROMloader.fname=$OUTDIR/mcp_romfw.bin \
	-C css.scp.ROMloader.fname=$OUTDIR/scp_romfw.bin \
	-C css.trustedBootROMloader.fname=$OUTDIR/$BL1_IMAGE \
	-C board.flashloader0.fname=$OUTDIR/$FIP_IMAGE \
	-C board.flashloader1.fname=$PWD/nor1_flash.img \
	-C board.flashloader1.fnameWrite=$PWD/nor1_flash.img \
	-C board.flashloader2.fname=$PWD/nor2_flash.img \
	-C board.flashloader2.fnameWrite=$PWD/nor2_flash.img \
	-S \
	-C css.scp.pl011_uart_scp.out_file=${MODEL_TYPE,,}/${UART0_SCP_OUTPUT_FILE_NAME} \
	-C css.scp.pl011_uart_scp.unbuffered_output=1 \
	-C css.scp.pl011_uart_scp.uart_enable=true \

Rebuild: 

~/rd-infra$ ./build-scripts/build-test-uefi.sh -p rdn2 all

Debug RDN2 

Find the start address of UEFI module 

TF-A loads BL33_AP_UEFI binary (the UEFI firmware) to the address specified by PLAT_ARM_NS_IMAGE_BASE in arm-tf/include/plat/arm/css/common/css_dev.h 

~/rd-infra$ grep PLAT_ARM_NS_IMAGE_BASE arm-tf/include/plat/arm/css/common/css_def.h

#define PLAT_ARM_NS_IMAGE_BASE U(0xE0000000)

The start address is 0xE0000000 

Start FVP 

Update MODEL environment variable with your FVP_RD_N2 model executable. 

export MODEL=~/FVP_RD_N2/models/Linux64_GCC-6.4/FVP_RD_N2

Go to model-scripts/rdinfra and run uefi.sh -p rdn2 

~/rd-infra$ cd model-scripts/rdinfra/ 

~/rd-infra/model-scripts/rdinfra$ ./uefi.sh -p rdn2 

The FVP will now start and wait for a debugger to connect. 

Create a debug connection 

Launch ArmDS Platinum 2021.a 

Create a new debug connection 

Select "Model Connection" and next

Enter a name for the connection.

Select "Add a new Model"

Select CADI as the model interface

Select "Browse for model running on local host"

Once your model terminal says “CADI Debug Server started for ARM Models...” the model should appear in arm DS.

Click Finish. The edit configuration window should then appear. 

Under “Connection” select Imported/Infra5_RD_N2/Bare Metal Debug/ARM_Neoverse-N2x15 Multi-Cluster SMP

Under Debugger, check “Execute Debugger Commands” and enter a breakpoint for the UEFI entry point we found earlier. 

Click Debug. The debugger should connect to the FVP. 

Get debugging script

The debugging script for UEFI is in the tianocore/edk2 repository so needs to be fetched manually.

~/rd-infra$ cd uefi/edk2

~/rd-infra/uefi/edk2$ git remote add tianocore https://github.com/tianocore/edk2

~/rd-infra/uefi/edk2$ git fetch tianocore

~/rd-infra/uefi/edk2$ git cherry-pick fddb8d24eccdcf467a8c777a9a22f4a1e54bf7dd

Load Debug Symbols 

If you have not connected to the FVP, click "Connect" to connect.

Currently it has stopped at EL3:0x00000000. This is the TF-A firmware. Click continue (F8)  to start execution. It will then break at EL2N:0xE0000000. 

Hit Continue (F8) again to let the FVP load into UEFI uninterrupted. This will load all the modules you would like to debug so that the debugger knows where they are.

Do the same actions in the same order you will when you are debugging as this affects the locations of the modules. For example, open UEFI interface -> Enter UEFI Shell -> Load EFI driver in the same order each time.

Take note of the file name of the UART log. This can be found in the terminal window where you ran uefi.sh. The UART log contains the addresses of the module locations and will be used to load the debug symbols.

Once you have loaded into the UEFI menu, stop the FVP by pressing Ctrl+C in the terminal window where you started the model, then press disconnect in the Arm DS window.

In Arm DS, go to the scripts tab and hit "Import a script or directory" then "Import a DS or Jython script"

Select the script "~/rd-infra/uefi/edk2/ArmPlatformPkg/Scripts/Ds5/cmd_load_symbols.py".

Right click the script in the list and select "Parameters"

Enter the following parameters

-p (<UEFI entry point>, 0x20000) The UEFI entry point was found earlier (0xE0000000). 0x20000 is the size which can be found in the UEFI build log.

-a -v

-i <path to UART log file> This is the path to the UART log which was produced by a running the whole UEFI process

-o <path to objdump executable> This is the path to aarch64-none-linux-gnu-objdump. If you followed the installation instructions, this will be ~/rd-infra/tools/gcc/gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-objdump

The script has now been setup.

Start the FVP again, hit the connect button in Arm DS and press continue into the UEFI firmware (0xE0000000). Load the debug symbols by right clicking the script and selecting "Run".

All the debug symbols have loaded, you can now start debugging!

Static vs Dynamic Modules

If you are debugging SEC or PEI you need to be aware of whether your module has a dynamic address.

Before PEI is reloaded, all the module addresses are defined at build time, however once the MMU has been initialised, modules are loaded and allocated memory. The dynamic addresses are the same between runs, however they may be different if you rebuild EDK2, or if you perform actions in a different order. If you are debugging a module before PeiCore has been reloaded, remove the -i and -o arguments from the cmd_load_symbols script so that the dynamic modules aren't loaded. The debug symbols for some modules will be defined twice if you don't do this, which could cause some symbols to not be loaded.

UEFI Boot Process

Now that the debug symbols are loaded, we can run through the UEFI boot process.

For more details on the boot process, check out the UEFI and PEI specifications.

Security (SEC)

First we enter the Security (SEC) stage. This is located where the branch instruction at the UEFI entry point (0xE0000000) is pointing. (0xE0017FD8).

SEC does some setup and passes control to PEI.

The module is located at uefi/edk2/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf. It is called ArmPlatformPrePeiCore.

Pre-EFI Initialization (PEI)

PEI sets up hardware abstractions of devices like Firmware and Memory for DXE to use. It does this searching the Firmware Volumes for PEIMs, loads them, then attempts to dispatch them.

When MemoryPeim is dispatched, which sets up the MMU, the PeiCore is reloaded into memory at a dynamic memory address.

PEI Foundation (PeiCore) entry point is located in uefi/edk2/MdePkg/Library/PeiCoreEntryPoint/PeiCoreEntryPoint.c

Pre-EFI Initialization Modules (PEIMs)

Pre-EFI Initialization Modules create an abstraction of a specific piece of hardware. They can work with other PEIMs using PPI (PEIM to PEIM Interface).

The PEI dispatcher calls one PEIM at a time, invoking it's entrypoint function. These functions perform initialization for the PEIM, which then register a service. This could be a PPI service for other PEIMs to call, or a HOB (Hand Off Block) which can be passed to DXE.

Now that all PEIMs are loaded and all corresponding HOBs have been created, PeiCore uses Dxe Initial Program Load (DxeIpl) with a list of HOBs to call DxeCore.

Drive Execution Environment (DXE)

The entrypoint of the DxeCore is DxeMain()

Similar to PEI, DXE traverses the Firmware Volumes looking for DXE drivers. It loads each driver and transfers control to them. DXE drivers can perform initialization, which could involve installing protocols which later stages can use. These drivers could be to enable input or output, or could be drivers for boot devices, like SD card or Network.

DxeMain initialises exception handlers, debug agent for DXE phase, DXE memory subsystem, EFI system table, EFI Runtime Services Table, Image services and Event services.

One of the drivers loaded is BdsDxe. This installs gEfiBdsArchProtocolGuid, which is used to jump to the BDS entry and transitions to the BDS stage.

Boot Device Selection (BDS)

BDSDxe reads the boot order list from EFI variables.

Boot timeout EFI is evaluated and user input from keyboard and mouse is connected. UI applications are shown.