Cortex-A7 initialization code & TrustZone/ Secure Boot

Hi vsiles,

It helps to dig into what the code in the _secure_monitor handler actually does - I took this paste from the original code provided by the RPi folks in your post:

1. _secure_monitor:
2.         mrc     p15, 0, r1, c1, c1, 0           @ read SCR
3.         bic     r1, r1, #0x4e                   @ clear IRQ, FIQ, EA, nET bits
4.         orr     r1, r1, #0x31                   @ enable NS, AW, FW bits
6.         @mrc    p15, 0, r0, c0, c1, 1           @ read ID_PFR1
7.         @and    r0, r0, #CPUID_ARM_VIRT_MASK    @ mask virtualization bits
8.         @cmp    r0, #(1 << CPUID_ARM_VIRT_SHIFT)
9.         orr     r1, r1, #0x100                  @ allow HVC instruction
11.         mcr     p15, 0, r1, c1, c1, 0           @ write SCR (with NS bit set)
13.         mrc     p15, 0, r0, c12, c0, 1          @ get MVBAR value
14.         mcr     p15, 4, r0, c12, c0, 0          @ write HVBAR
16.         @ Reset CNTVOFF to 0 before leaving monitor mode
17.         mov     r0, #0
18.         mcrr    p15, 4, r0, r0, c14             @ Reset CNTVOFF to zero
19. 1:
20.         movs    pc, lr                          @ return to non-secure SVC

The "orr" on line 4 of the paste above is setting the bottom bit in a temporary register, which is intended to go into the SCR (line 11).

There seems to be a little confusion on what SCR.NS actually does - it is prudent to note that the bit does not actually change the security state of the processor (Monitor mode is always a Secure state), but it does provide a way to access what the ARM Architecture Reference Manual refers to as the Secure Banked CP15 Registers and their Non-Secure Banked CP15 Registers equivalents. In this case the moment the SCR register is written at line 11, the intent of the code is that you want to access the Non-Secure copies. If we proceed to line 14 we see that it is intending to write to the HVBAR. The HVBAR is a special register in that it isn't part of the Secure Banked CP15 Registers and is only present in the Non-Secure set. CNTVOFF has very similar restrictions - it is only accessible when SCR.NS=1. No matter what the state of the NS bit, if you are in Monitor mode, you're in Secure state.

There's a good description of the banking and register accessibility on page B3-1459 of the ARM ARM (DDI 0406C.c). You can get a copy here: ARM Architecture Reference Manual ARMv7-A and ARMv7-R edition

There is a small spanner in the works, however, in that the effect of the write to SCR at line 11 only really takes effect after a context synchronization event - the ARM ARM defines what these are, but for now we'll just say that you either need some kind of exception return or entry (i.e. a mode change), or an explicit ISB instruction. Until that synchronization occurs, any instructions after the write may respect either value of the SCR.NS bit, depending on the state of the context. It isn't possible to know which instruction will see the effects of the SCR.NS=1 write unless such synchronization has happened. The discussion on synchronization of changes to system control registers directly follows the abovementioned description (B3-1461).

To be technically correct, the code should include have an ISB after the SCR write to ensure that the HVBAR and CNTVOFF writes will succeed (yes, in the original code, too), and the ISB before the "smc #0" is redundant (since it will perform the same action). Otherwise it could simply and randomly cause an unwanted exception, and since the MVBAR vector table isn't complete, it will spiral down the toilet -- the mcr will be an undefined instruction, and the undefined instruction exception vector is .word 0, which causes another undefined instruction exception, ad infinitum.

In terms of the requirement to use "smc #0" in Cortex-A7 initialization, this isn't necessary at all - it seems to be a concession to code size for this small bootstrap, and to allow future software such as U-Boot to execute that code path and return to it. The code itself seems to simply be a way to put the core (whichever it is, since there are four) in a good state and then loop forever until somehow given something else to run, or jump to 0x8000 if it's core #0.

One thing I am curious about is how you know that the CPS instruction is the cause or if you see the place it hangs (or the symptoms). Do you know that you are not in Monitor mode already? Do you have a debugger attached? I have to say I am a little confused as to exactly where your code is loaded, whether the RPi team's code is still loaded to 0x00 - 0xFF, and whether you're basically entering this from the U-Boot start.S code (when it calls "smc #0")? If your code runs after the RPi code?

Ta,

Matt Sealey

In the _secure_monitor, if I do not set SCR.NS to 1, I expected to stay in secure mode, but the boot hangs

Does the default board setup actually set up any secure resources?

In "normal" operation the system is flipped over to non-secure during boot, so I would expect most of the memory and peripherals to also have been configured as non-secure so that Linux / Android / etc is able to use them. In your case you need some secure resources to be available to the software (or secure page tables configured to access non-secure memory areas - although this doesn't carry any security benefits).

HTH,
Pete

Hi, thank you for the detailed feedback.

I'll start with your question first: The "normal" configuration of the raspi is to run this code (which is linked at 0x0000) and then jump core 0 at 0x8000 where uboot/kernel/random code should have been linked. Using a config option, one can disable this behavior and make the board boot by executing code at 0x0, in monitor mode. In this configuration, no initial boot code from raspberry is executed. For my tests, I used this second booting option, linked their code at 0x0, and my code at 0x8000. Therefore, by modifying their code, I can see the changes once (if) I reach my own code. The issue with this approach is that I don't have any debugger, I use uart output to debug, thus the difficulty to debug this part of the boot code. I am really not sure where the board hangs during the boot, but I (wrongly) guessed it had to do with changing mode.

I think my main mistake was about the NS bit. I thought that you can only accessed the banked Secure CP15 register while in monitor mode, because monitor mode is always considered secure. If I understand correctly your explanation, I should add a ISB after line 11 so that I am sure to write to the correct version (non secure) of the CP15 registers. So if I want to exit in secure mode, I have to switch back to NS = 0 "after" this CP15 changes ? I'll try that today and come back to you with more feedback.

Best,

Vincent

To my knowledge, the board does not initialize anything except for this small boot code that you can disable if you want (see my answer to Matt for the details). In this configuration, I am sure to boot in monitor mode with the MMU turned off.

I think I finally understood what happens and I managed to do exactly what I wanted in the "boot" part of their code.

Just for the record, I added an ISB in the secure monitor, to be sure. And after changing the "non secure only" cp15 registers, I switch back NS to 0. Then, when I exit from the smc call, I am still in SVC secure mode.

I am now able to rewrite this boot sequence in another way, and setup a meaningful secure_monitor instead.

Thanks to your input, it was very helpful.

Best,

Vincent