Bare metal development on Juno

Introduction

Unlike some earlier development platforms that allowed easy baremetal access to the SoC, Juno is a complex system that requires a certain level of initialisation even before a baremetal application can be run. It's perhaps best to think of the Juno as being similar to a traditional desktop PC; the SMM firmware and BIOS/UEFI are always present and you can only take control after they've run.

With that said, it is possible to safely take control of the Juno at EL3 with Secure world privileges but it requires reconfiguring of various protections and for care to be taken not to trample over certain areas of reserved memory.

So, before beginning, some fundamental knowledge of the Juno SoC architecture and Arm Trusted Firmware boot process is required.

Juno SoC architecture

The Juno SoC is comprised of a "big" cluster (Cortex-A57 MP2 on r0 and r1, Cortex-A72 MP2 on r2), a "LITTLE" cluster (Cortex-A53 MP4), and a "system control processor" (Cortex-M3 microcontroller).

The system control processor (shorthand "SCP") is responsible for managing the power and clocks of all other SoC components, including voltage regulation and thermal protection of the A-class processors.

Arm Trusted Firmware boot process

The Arm Trusted Firmware (shorthand "ATF") boot process spans multiple individual boot stages.

`SCP_BL1' is the ROM firmware image running on the SCP at reset.
This is a minimal bootstrap image that lacks dynamic voltage regulation and thermal protection of the A-class processors.

The intention is for this image to be able to boot an A-class processor far enough that it can load a more complex runtime firmware image from Flash and "transfer" that image to the SCP.
`AP_BL1' is the ROM firmware image running at EL3 on the A-class processors at reset.
This is a minimal bootstrap image that would be stored in ROM on production systems but is executed directly from Flash on Juno due to it being a development platform (allowing us to reflash the image with updates).

The boostrap loads a more complex bootloader (`AP_BL2') into Secure memory, authenticates that image using certificates, then passes control to it at Secure EL1.
`AP_BL2' is a bootloader that loads the Juno's runtime firmware image (`AP_BL31') into Secure memory, the SCP's runtime firmware image (`SCP_BL2') into Secure memory, the Trusted OS kernel e.g. OP-TEE (`AP_BL32') into Secure memory, the Normal world bootloader image e.g. UEFI or U-Boot (`AP_BL33') into Non-secure memory, and authenticates all of these images using certificates.
The bootloader then "transfers" `SCP_BL2' to the SCP via a defined comms channel; from this point onwards we have full voltage regulation and thermal protection of the A-class processors.

Finally the bootloader returns control to `AP_BL1' at EL3 by executing an `SMC' instruction.
`AP_BL1' reclaims the Secure memory that was occupied by `AP_BL2' then branches to where `AP_BL31' was loaded.
`AP_BL31' is the runtime firmware image that remains resident at EL3 on the A-class processors even after control has been passed to the Normal world.
This image acts as the Secure Monitor handling switches between the Normal world and Secure world, and also handles runtime firmware requests from the Normal world such as for hotplugging A-class cores and entering subsystem deep idle power states.

During the boot process `AP_BL31' passes control to the Trusted OS kernel image (`AP_BL32') at Secure EL1.
`AP_BL32' is the Trusted OS kernel image e.g. OP-TEE.
This kernel image remains resident at Secure EL1 and handles Secure requests from the normal world, see this page on the Linaro wiki.

After initialising itself the Trusted OS kernel returns control to `AP_BL31' at EL3 by executing an `SMC' instruction.
`AP_BL31' performs a world switch and passes control to the Normal world bootloader (`AP_BL33') at EL2.
`AP_BL33' is the Normal world bootloader e.g. UEFI or U-Boot.
At this point the Normal world bootloader performs its regular duties, for example booting the Linux kernel.

In addition to not having voltage regulation or thermal protection of the A-class processors until after `AP_BL2' has "transferred" `SCP_BL2' to the system control processor, each of the `AP_BLx' images performs its own platform-specific initialisation on the Juno.

We therefore need to wait until at least `AP_BL2' has returned control back to `AP_BL1' (but ideally we will allow all of Arm Trusted Firmware to run and only take control of the Normal world).

Prerequisites

Latest version of DS-5 Development Studio
Suitable JTAG debugger such as a DSTREAM

Taking control of the Juno

Follow the instructions here to install an EDK2-based UEFI prebuilt configuration on your Juno.

Once installed, power cycle the Juno; you should be dropped into an EDK2 shell.

In DS-5 create a new DS-5 Debug Configuration for the Juno:

Create a new launch configuration
Select this target, replacing `r0' with your Juno revision:
- Juno Arm Development Platform (r0)
  - Bare Metal Debug
    - Debug Cortex-A53_0
Browse for your DSTREAM in the `Connections' section
On the `Debugger' tab, under `Run Control' select `Connect only'

You should now be able to use this configuration to connect to the Juno and interrupt the primary CPU using the debugger GUI; check that you're in EL2 then continue to the next section.

Compiling a baremetal image

When compiling a baremetal image targeting Juno you'll need to ensure it is loaded to a valid location in memory. The Juno's A-class processor memory map has two regions of DRAM:

2GB at [0x0080000000 - 0x00FFFFFFFF]
6GB at [0x0880000000 - 0x09FFFFFFFF]

You can use a simple scatter file to link your baremetal image into one of these regions, for example:

LOAD 0x80000000
{
    DRAM 0x80000000
    {
        * (+RO, +RW, +ZI)
    }
}

Running your baremetal application

How you proceed will depend on whether your baremetal application runs in the Normal world (EL2) or in the Secure world (EL3).

We strongly recommend running your baremetal application in the Normal world (EL2) if possible. This means you'll still have the Arm Trusted Firmware runtime firmware image resident at EL3 and can therefore make Power State Coordination Interface (PSCI) calls to power cores up and down. As detailed below it also drastically simplifies the process of taking control of the Juno.

EL2

Execute these commands in the debugger command window:

interrupt
core apply all set var $AARCH64::$System::$Other::$SCTLR_EL2 = 0x0
core apply all set var $AARCH64::$System::$Virt::$HCR_EL2 = 0x0
set var $AARCH64::$System::$PSTATE::$DAIF.D = 1
set var $AARCH64::$System::$PSTATE::$DAIF.A = 1
set var $AARCH64::$System::$PSTATE::$DAIF.I = 1
set var $AARCH64::$System::$PSTATE::$DAIF.F = 1
set var $AARCH64::$System::$PSTATE::$Mode.M = 9

You can also paste these into the `Execute debugger commands' section of the `Debugger' tab when editing the DS-5 Debug Configuration created earlier; this way you avoid having to type them out each time you connect to the board.

These commands will:

Ensure that the MMU is disabled at EL2
Clear `HCR_EL2' to known safe value
Mask all asynchronous exceptions (Debug, SError, IRQ, FIQ)
Force the CPU to enter EL2h mode

Next skip to loading a baremetal image on Juno.

EL3

The Juno SoC includes a TrustZone Address Space Controller (TZC-400) that is used to make certain areas of DRAM only accessible to the Normal world and other areas only accesible to the Secure world.

By default the TZC-400 is configured like so:

[0x0080000000 - 0x00FEFFFFFF] --> Non-secure ONLY
[0x00FF000000 - 0x00FFFFFFFF] --> Secure ONLY (reserved for SCP *)
[0x0880000000 - 0x09FFFFFFFF] --> Non-secure ONLY

* And the Trusted OS kernel e.g. OP-TEE if using Arm Trusted Firmware's option for `AP_BL32' to reside in TZC-protected DRAM.

By default all accesses made at EL3 are Secure; the only way to make Non-secure accesses is by enabling the MMU and setting the translation table entry descriptor `NS' bits, which means we cannot simply load the baremetal image into DRAM and run it because the MMU will be disabled and thus the TZC-400 will block the memory accesses.

There are two options to work around this:

Use the debugger to reprogram the TZC-400 to allow Secure accesses to DRAM
Recompile Arm Trusted Firmware using its EL3 payload option

We recommend the second option but have documented both here.

Reprogramming the TZC-400

The TZC-400 defines a number of regions that have a base address, limit address, and set of attributes. On the Juno region 1 is the Secure ONLY [0x00FF000000 - 0x00FFFFFFFF] range reserved for the SCP.

The simplest way to reprogram the TZC-400 to allow Secure accesses to all of DRAM is therefore to lower the base address of region 1 to `0x80000000' by reprogramming the `REGION_BASE_LOW_1' register. This is at address `0x2A4A0120' on the Juno.

WARNING: You must ensure your baremetal image does not modify the contents of the region [0x00FF000000 - 0x00FFFFFFFF] as the SCP still expects to have this region reserved for itself. Remember that the SCP is responsible for voltage regulation and thermal protection of the A-class processors; trampling over the state that it keeps in this area of DRAM is potentially dangerous. Also if using Arm Trusted Firmware's option for `AP_BL32' to reside in TZC-protected DRAM, writing to this area of memory will be trampling over the Trusted OS kernel image resident at Secure EL1.

Execute these commands in the debugger command window:

interrupt
core apply all set var $AARCH64::$System::$Other::$SCTLR_EL3 = 0x0
core apply all set var $AARCH64::$System::$Secure::$SCR_EL3 = 0x0
set var $AARCH64::$System::$PSTATE::$DAIF.D = 1
set var $AARCH64::$System::$PSTATE::$DAIF.A = 1
set var $AARCH64::$System::$PSTATE::$DAIF.I = 1
set var $AARCH64::$System::$PSTATE::$DAIF.F = 1
set var $AARCH64::$System::$PSTATE::$Mode.M = 13
memory set_typed AXI<PROT=1>:0x000000002A4A0120 (unsigned int) 0x80000000

These commands will:

Ensure that the MMU is disabled at EL3
Clear `SCR_EL3' to known safe value
Mask all asynchronous exceptions (Debug, SError, IRQ, FIQ)
Force the CPU to enter EL3h mode
Reprogram the TZC-400's `REGION_BASE_LOW' to `0x80000000' to allow Secure accesses to lower region of DRAM

Next skip to loading a baremetal image on Juno.

Recompiling Arm Trusted Firmware

Arm Trusted Firmware has the ability to pass control to an EL3 payload, i.e. a baremetal image that we've already loaded into memory.

This is done using two compile-time flags:

`EL3_PAYLOAD_BASE' points to the entrypoint of your baremetal image.
Enabling this option prevents the need to supply a Normal world bootloader (`AP_BL33') and the resulting Firmware Image Package (FIP) that gets flashed to the board will not contain `AP_BL31' either as the resident runtime firmware is never reached.
`SPIN_ON_BL1_EXIT', when used in conjunction with `EL3_PAYLOAD_BASE', will force ATF to spin forever just before it would have branched to the address specified by `EL3_PAYLOAD_BASE', thus giving you the opportunity to connect a debugger and load your baremetal image into memory.

To compile ATF using these options:

$ export CROSS_COMPILE=/path/to/aarch64/cross/compiler/gcc/bin/aarch64-linux-gnu-
$ make PLAT=juno DEBUG=1 SCP_BL2=scp_bl2.bin EL3_PAYLOAD_BASE=0x80000000 SPIN_ON_BL1_EXIT=1 fip all

Where `scp_bl2.bin' can be found in the `<workspace>/juno-uefi/SOFTWARE/' folder after using the workspace initialisation script to sync an EDK2-based UEFI prebuilt configuration as described earlier.

Note that this flag and corresponding binary were originally named `BL30' and `bl30.bin'; Arm Trusted Firmware renamed the flag to `SCP_BL2' but there was a period of time before the Linaro releases renamed the binary to `scp_bl2.bin'. If you encounter any of this outdated nomenclature, ensure you're using the latest deliverables by using the workspace initialisation script linked above.

Loading a baremetal image on Juno

After connecting to the Juno and interrupting the primary CPU:

From the `Debug Control' view, navigate to `View Menu' > `Load...'
From the dropdown select `Load Image and Debug Info'
Navigate to your compiled baremetal image
Leave the `Load Offset' field blank
How you proceed will depend on whether your recompiled Arm Trusted Firmware:
If you did not recompile Arm Trusted Firmare:
1. Tick the `Set PC to entrypoint' checkbox
2. Press `OK'
If you did recompile Arm Trusted Firmware:
1. Press `OK'
2. Enter the following debugger command:
```
set $pc += 4
```
3. Set a hardware breakpoint on the entrypoint of your baremetal image, for example if your entrypoint is `0x80000000':
```
hb 0x80000000
```
4. Resume the primary CPU; Arm Trusted Firmware will finish initialising and then branch to your baremetal image, hitting the hardware breakpoint set above
You can now run and debug your baremetal image

Using Power State Coordination Interface (PSCI)

Arm Trusted Firmware implements version 0.2 of the Power State Coordination Interface (PSCI), allowing software running in the Normal world to make `SMC' instruction calls into the runtime firmware resident at EL3 in order to make power management requests, for example to hotplug cores in/out of the system and to enter subsystem deep idle power states.

To request power up of another core you can use the `CPU_ON' function which takes four arguments:

`X0' -- PSCI function ID of `CPU_ON'
`X1' -- `MPIDR_EL1' affinity value of the core to power on (non-`Aff' fields zeroed)
`X2' -- Entrypoint address i.e. where Arm Trusted Firmware will `ERET' to after initialising the CPU
`X3' -- This contextual value will be in `X0' at the point that Arm Trusted Firmware performs an `ERET' to the specified entrypoint

So for example to power on Cortex-A53_2 (`Aff3' =0x00, `Aff2' =0x00, `Aff1' =0x01, `Aff0' =0x02):

LDR     X0, =0xC4000003
LDR     X1, =0x102
ADRP    X2, secondary_entrypoint
ADD     X2, X2, :lo12:secondary_entrypoint
MOV     X3, XZR
SMC     #0

The Arm Trusted Firmware runtime will program the specified entrypoint and contextual value into a set of memory-mapped registers and request the SCP to power on on the specified CPU; it then returns control back to the Normal world.

When the SCP releases the specified CPU from reset, `AP_BL1' reads from the memory-mapped register holding the specified entrypoint and gets a non-zero value indicating that this is a warm boot; most of the regular Arm Trusted Firmware boot stages are skipped and the CPU instead jumps straight into the runtime firmware that is already resident in memory. The runtime firmware initialises the CPU, places the specified contextual value in `X0', and performs an `ERET' to the value read from the memory-mapped register.

See the PSCI documentation linked above for full function prototypes.

Recommended