Hi,
I have a system with a multiple quad core clusters with Cortex A-53 and the CCN-512. L1 through L2 are integrated caches where L3 is an outer cache in 8xHN-F of the CCN512.
My question is how should I interpret the shareability domain: inner, outer controlled through TCR register and the page descriptor. Would be that inner shareable is within the Cluster and only within it the Coherency is maintained for the Memory region marked Cacheable WB-WA for Innter and Outer?
Or is my understanding wrong in the context of "Snoop and Maintenance Requests" chapter of "ARM Cortex-A53 MPCore Processor TRM". It says there that broadcastinner asserted enforces broadcastouter asserted so that would suggest that setting the inner shareability makes snoop and maintenance requests boradcast to the Observers in both Inner and Outer Domains.
Please help!
Thanks,
Marek
Raghu, thank you a lot for the in detail elaboration. My comments are:
1) Yes I know the shareability is a SoC not Processor specific. I also have cortex a57 with CCN0504 and also L3 in there but wanted to go processor specific in the question
3) I cannot provide more details. This is all I have. I have a system with 8x Quad Cortex a53 and 4x Ceva Clusters with CCN512 as Coherent Interconnect. 8x HN-Fs (Fully Coherent Home network) host all together 24M of L3 cache with 3M each. And as per CCN-512 each HN-F manages PoC and PoS, tracks HN-F caching in the snoop filter. Snoop filter tag RAM is 4M.
5) Read that too and that confused me even more as that would suggest that if all cores run the same OS then L1 through L3 would be in the same inner shareability domain. But on top of that I read in “ARM® Architecture Reference Manual ARMv8, for ARMv8-A architecture profile”
Example B2-1 Use of shareability attributes In an implementation, a particular subsystem with two clusters of PEs has the requirement that: • In each cluster, the data caches or unified caches of the PEs in the cluster are transparent for all data accesses to memory locations with the Inner Shareable attribute. • However, between the two clusters, the caches: — Are not required to be coherent for data accesses that have only the Inner Shareable attribute. — Are coherent for data accesses that have the Outer Shareable attribute. In this system, each cluster is in a different shareability domain for the Inner Shareable attribute, but all components of the subsystem are in the same shareability domain for the Outer Shareable attribute. A system might implement two such subsystems. If the data caches or unified caches of one subsystem are not transparent to the accesses from the other subsystem, this system has two Outer Shareable shareability domains.
That would suggest that the design above is that Cluster#0 runs one OS whereas Cluser#1 another. Isn't it?
6) This is also my understanding and I would infer from that that if in any doubts I'm better off to set outer shareability as than the coherency will get maintained for the inner (L1/L2) and outer shareable (L3) domains.
For 5) yes. In the example system described in Example B2-1, the different clusters would typically, but not mandatory, be running different OS's. The document is only referring to expected use case but the architecture does not explicitly forbid running multiple OS's running in the same inner shareable domain.
For 6) Even If you mark your memory as outer shareable, there is no guarantee that coherence "Will" get maintained. The SOC hardware may not even have(or recognize, or may simply force that OSH = ISH) an outer shareable domain. It is possible that the SOC is designed in such a way that coherency between the clusters is expected to be maintained manually. You may want to mark your memory as outer shareable and experiment to see if coherency is maintained but the safest approach to designing software for the SOC would be to know exactly how the SOC is designed to avoid ending up with hard to debug coherency issues. In your case, as stated earlier, it is possible that L1/L2 are inner caches and L3;s are outer caches but are all part of the inner shareable domain in which case you can just mark all your memory as innershareable and not have to worry about coherency between clusters.
Great. Thank you Raghu. This is invaluable information. I have requested the HW design team to let me know these.