TLB coherence expectation with TF-A running on ARM Neoverse N1 cores and CMN-600 network

Hi,

I am running some bare-metal TF-A based tests for coherence on a multi-PE system with multiple ARM Neoverse N1 cores connected through a CMN-600 mesh network.

I tried the following sequence, in order to test the TLB behavior at EL3 across multiple PEs - 

1) Ensure that the TTBR0/1_EL3 of the MMUs in all PEs are pointing to the same location - ie. the same set of translation tables is being used by all PEs.

2) Obtain a physical address with RO attributes

3) Read this location from all PEs in order to get the associated translation table entry into the TLBs at all the PEs

4) Change the MMU attribute of this page to RW using the TF-A - xlat_change_mem_attributes() library function. This function performs the necessary attribute change and also executes the TLBI -> DSB ->ISB sequence needed to invalidate the cached and stale TLB entry and make the effect visible at the modifying core

       i. My idea of TLB coherence, is that the TLBI operation executed on the attribute modifying core should automatically lead to the cached and stale TLB entries at the other PEs to also get invalidated. Is this the correct expectation?

5) To verify that all the PEs are able to observe the modified attribute, I perform a write to the modified page from all the PEs, expecting the write to go through without causing an exception.

However, I see that unless the xlat_change_mem_attributes() function (with the associated TLBI sequence) is executed on all the PEs, the writes performed by the other PEs cause a Permission fault.

Please could you provide some information on why this TLBI sequence needs to be executed on all the PEs in spite of the hardware assisted TLB coherence mechanism being present?

Thanks

Ganesh Garga