Hi everyone,
I'm currently working on type-1 hypervisor and would like to provide support of the ARM Power State Coordination Interface. http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/Power_State_Coordination_Interface_PDD_v1_1_DEN0022D.pdf
In order to achieve this, some functionality should be implemented in EL2 mode in response to the hypercalls performed by the guest OSes. According with the Platform Design Document (above) some functions are mandatory such as CPU_OFF, CPU_SUSPEND and CPU_ON. However, in both of them it is required cache and coherency management.
My question is the following, in a scenario that there are multiples vCPUs running on top of physical core0 and one of them wants to perform a hypercall to the hypervisor (CPU_ON), is there any portable way to invalidate the respective cache lines related to the specific vcpu, the one which performs the hypercall?
Thanks in advance,
Cheers.
Jorge
Jorge,
The cache maintenance requirement is for the purpose of maintaining system coherency over a power down event. If you pull power from the cache RAM arrays they'll lose their data - so cleaning them to a point which will remain powered is necessary, otherwise whatever was still dirty in those arrays (i.e. not consistent with the lower levels of memory) is lost forever.
Hypervisors shouldn't need to touch the caches in this instance, unless they truly are going to power down the core, in which case the actual PSCI will handle it for you.
Ta,
Matt
Hi Matt,
Thanks for your prompt response.
I will try to make it clearer, basically, in my setup I have two guests running concurrently, each one on its vCPU. Guest0 in vCPU0 and Guest1 in vCPU1. Those vCPUs are an abstraction of the physical CPU0. All PSCI resquests (CPU_ON, OFF, SUSPEND) from guest OSes in their vCPUs will be trapped by the hypervisor.
Also, each vCPU has a state which signalizes the vPSCI state: RUN, STANDBY, RETENTION or POWERDOWN.
As you correctly said, my hypervisor will touch the cache using the actual PSCI presented in ATF only if all vCPUs (Guest0/vCPU0, Guest1/vCPU1) running on top of the physical CPU0 are in a state like POWERDOWN.
Otherwise, if the vPSCI of the guest0/vCPU0 is in running state and the Guest1 performes a hypercall to powerdown its virtual instance of CPU0 (i.e. vCPU1) , I would like to flush the all caches lines related to the vCPU1, since the vCPU1 is going to powerdown but the physical one remains running to run the vCPU0.
Did you get my point?
Thanks in advance.
I'm not sure I do get it.
so vCPU0 of Guest 0 on CPU0 is running. vCPU0 of Guest 1 on CPU 1 does a vPSCI call to "power down" but the hypervisor will not power down this core via PSCI.
You want to clean all the lines out of CPU 1 for Guest 1 even though it will stay in a physical running state.
Unless your guests are sharing data there's no need - are you attempting to make sure vCPU 0 for "Guest 2" has an empty cache?
The reason PSCI requires cache maintenance is not only to ensure consistency of the data used by the OS but to cover Secure data which cannot be actively maintained by the Non-Secure world (this is why PSCI lives at EL3 or EL3+EL1S.) It also makes sure that if a core is in a state that cannot respond to coherency traffic that the interconnect (or SCU) does not send snoops that will never be acked.
For two Non-Secure guests under virtualization unless you are going to actually place either physical CPU in a low power state (I.e. Call PSCI at EL3) then natural processes will ensure things stay coherent and consistent. Active cache maintenance at the point of a Virtual powerdown is a waste of power and bandwidth, and if it does power down via PSCI then it will do that for you anyway.
So, again, I'm not sure I understand the intent you have.. but I still don't think you need to do anything at all. If you still think you need to then DC CVACxx over the range will do it (clean by VA, to point of coherency).
When the vCPU powers up again if you need to be sure you didn't prefetch anything in the meantime then you'll need a DC IVACxx on that side, no different from any other Shareable memory location (there's fundamentally no difference between two cores and one core and a DMA engine in the memory model or AMBA architecture, consistency between two masters is /just/ consistency between two masters, and it's managed for you by the cache coherency protocol).
Thank you so much.
That's what I wanted to know -
"The reason PSCI requires cache maintenance is not only to ensure consistency of the data used by the OS but to cover Secure data which cannot be actively maintained by the Non-Secure world (this is why PSCI lives at EL3 or EL3+EL1S.) It also makes sure that if a core is in a state that cannot respond to coherency traffic that the interconnect (or SCU) does not send snoops that will never be acked.
For two Non-Secure guests under virtualization unless you are going to actually place either physical CPU in a low power state (I.e. Call PSCI at EL3) then natural processes will ensure things stay coherent and consistent. Active cache maintenance at the point of a Virtual powerdown is a waste of power and bandwidth, and if it does power down via PSCI then it will do that for you anyway."
thank you once again.