Given a multiprocessor system, how are the PC values of secondary cores set from the primary core? I've read lots of threads stating it can be done but without any details. I could not find anything in the ARMv8 reference manual.
One might contrive a mailbox system where the secondary cores wfe and check the value of a mailbox address when woken up. But is there a supported method to directly write the secondary core PC before waking it up? What are the various methods for "waking up" a core?
Hi Branden Sherrell,
As an example, Linux permits two different options for waking up secondary cores on "arm64":
- Spin table (write PC to memory)
- PSCI (call hypervisor/firmware)
See Documentation/arm64/booting.rst
Starting the secondary cores depends on the SoC. Some have a dedicated register which contains the start address. On some the core just starts at 0 and you have to store there a jump to the actual boot code for the secondary core.
So, you have to check you SoC's manual.
But you cannot write any of the registers of another core.
I should have clarified I meant post-boot. Once the primary core has initialized the system I was wondering how secondary cores have their PC set.
Thanks for the link! DEN0022A is mentioned in that documentation. I took a look at the PCSI monitor call for CPU_ON but it is described only in a way that one might describe a function call. The implementation seems to be undefined. That said, it seems to be a standardized method for pointing a CPU to an entry point, but its implementation might use a spin table, for example.
It seems overly complicated, in my opinion, to rely on EL3 to enable a CPU. I say that because you will be issuing the SMC from another core. So during the exception handling you would need to signal the available CPU to _also_ execute an SMC instruction so that it can ERET to the appropriate exception level, right? What about CPU_OFF or CPU_SUSPEND? What hardware support is available to implement these calls? This seems to be describing control of secondary CPUs from a primary (or just another) CPU, but you mentioned it was not directly possible?
Maybe the trusted firmware has an implementation example for these. It seems like a hard synchronization problem at first glance. Perhaps I am missing something. Is there an event that can be sent to all cores to send them to EL3 for such a synchronization even? It seems then that with MPIDR one could check its ID and eret to the appropriate PC/EL while all other cores exit immediately.
There is no other way.
Of course you can then setup some kind of mailbox these cores check.
... and maybe, be more specific about which SoC you are talking.
I've been prototyping with QEMU. I had no idea something like this might be SoC-specific. I've seen entrypoint address registers in some NXP SoCs but I was not sure what was standardized for this kind of thing.
The PSCI reference made above is an interesting one. After reading into the documentation I find it confusing how actions such as CPU_ON/OFF and SUSPEND can be had through an SMC if cores are unable to influence the PC of other cores. Unless they are not considering the PC and just using some other means to shut the core down? Can that be done?
PSCI is AFAIK part of the trusted firmware which runs below all other software. So when you tell it to "start" a CPU it will take the actions described before. Only, the started CPU will boot into the trusted firmware and then jump into the user code.
The trusted firmware is always present on each running core. A CPU_OFF on one core will likely send a message (via IPI) to the other one to handle this.
But the basic principal is the same: A core starts from a fixed address. Often 0, sometimes (high vectors) 0xfff0.0000. On iMX8, you have to set a SoC register with the start address. But it is likely that the started core first executes ROM code in the SoC and then the ROM code jumps to the address in the SoC register.
But I do not know how QEMU handles this. Maybe all cores are started but loop and check a specific memory address for a start address.
It depends on how you are using QEMU. I've been using it in such a way that I'm writing the "firmware" for the emulator. My code is mapped into the Flash region and each core starts executing the first instruction (address zero) on reset. So it's entirely up to me I suppose for how this is done.
Thanks for the info.
It took me a bit of poking around to finally uncover that IPIs on ARM are referred to as LPIs, for anyone here after me that might be searching for the same thing. Implemented in the GIC.
Hi 42Bastian Schick,
In my case I see qemu emulates PSCI called with HVC.
You can ask qemu to dump the dtb with:
qemu-system-aarch64 -machine virt,dumpdtb=dump.dtb -smp 4 ...
Then you can "disassemble" the dtb with:
dtc -o dump.dts -I dtb -O dts --sort dump.dtb
You will see the cpu nodes in there referring to PSCI:
cpus { #address-cells = <0x01>; #size-cells = <0x00>; cpu@0 { compatible = "arm,cortex-a72"; device_type = "cpu"; enable-method = "psci"; reg = <0x00>; }; cpu@1 { compatible = "arm,cortex-a72"; device_type = "cpu"; enable-method = "psci"; reg = <0x01>; }; cpu@2 { compatible = "arm,cortex-a72"; device_type = "cpu"; enable-method = "psci"; reg = <0x02>; }; cpu@3 { compatible = "arm,cortex-a72"; device_type = "cpu"; enable-method = "psci"; reg = <0x03>; }; }; ... psci { compatible = "arm,psci-0.2\0arm,psci"; cpu_off = <0x84000002>; cpu_on = <0xc4000003>; cpu_suspend = <0xc4000001>; method = "hvc"; migrate = <0xc4000005>; };
Best regards,
Vincent.
Thanks for the link!