question on Exclusive access

Martin has hopefully answered your first question, although a simpler way of answering might simply be to state that exclusives are there to support read-modify-write sequences, so a read is always the first step in the sequence.

For your second question, the exclusive access monitor is not part of the interconnect. It is usually part of the destination target device, so any write response the monitor returns will need to factor in both whether the exclusive nature of the transfer completed successfully or not, and also the OKAY/ERROR nature of the transfer.

So if the exclusive write was a "non-bufferable" write, the response returned on BRESP will indicate if the transfer completed exclusively, and also if the transfer was stored in the "S0" destination.

I said above that the monitor is "usually" part of the destination device. This can be fully integrated with it, or perhaps a standalone monitor attached to the destination device. But it could also be further away from the final destination device, as long as it is able to see all transfers that can access the final destination device.

Hi Martin, Thanks for reply.  You said The exclusive write happens only if the tag is still there. if Master already tagging the address, why the tag is missed? Can other Masters delete the tag?

Hi Colin, thanks for reply

if Master use non-bufferable write, the performance are usually degraded(because of bresp must be obtained from the final destination)? 

The exclusive access monitor watches for successful writes, or possibly other exclusive reads, from other masters in the system to the monitored address.  

Let's say you have two CPUs trying to down a counting semaphore.  Both need to atomically test whether the semaphore is greater than zero, and if it is, decrement its value in memory.

Both CPUs will try to something along the lines of LDREX, SUBS, BMI, STREX, CMPS, BNE in a loop.  SUBS will detect 0 by the result of the subtract going negative, and set N=1 if it does.  BMI is to branch to a "go to sleep, semaphore is busy" path.  The CMPS, BNE will loop if the STREX failed.  CMPS, BNE may be replaced with CBZ/CBNZ if available.

Imagine the following interleaved accesses to the semaphore by the two CPUs:

In this scenario, CPU A was blocked from updating the semaphore, because B raced with it and snatched the exclusive monitor away from it.  Now, notice I said "successful writes."  Because the STREX was denied, there's no reason for it to redirect the monitor.

The monitor can implement a different policy.  If a second CPU tries to obtain the monitor for the same address before the first CPU has made its write, the monitor can decide not to switch, like so.

The second strategy has some advantages if you guarantee LDREX/STREX pairs are guaranteed to arrive close to each other, or you place a timeout on the monitor.  If you get an interrupt or a long stall between LDREX and STREX, you could cause undue fairness issues in the system.  Note: CLREX can be used to clear active monitors in an interrupt handler.

For device and non-cacheable memory, exclusive access monitors are typically implemented in the endpoint device, or at an upstream point in the interconnect that all accesses to that endpoint must come through.  Home Nodes, for example, in CHI interconnects, could implement a monitor for downstream memory or devices.  Or, you could imagine a proxy bridge in AXI interconnects that could work with arbitrary IP behind it.

For normal memory in a coherent interconnect, the L1 cache controller can implement the exclusive monitor for normal, cacheable, shared address ranges, by pulling the line into the Exclusive state.  Coherence snoops due to other CPUs' accesses would cancel the monitor.  That strategy looks like the first scenario above.  I believe this is what the Cortex-A53 does with its internal exclusive monitor.  (See the TRM.)

Note: My description above is meant to be generic for this class of primitive, and not specific to a particular ARM implementation, except where noted.  Other platforms call this primitive "Load Link, Store Conditional." That may help you find other resources on how this form of optimistic synchronization primitive works.

Yes, that's definitely true.

But in your question when you were assuming the monitor was in the interconnect, you were concerned that the access hadn't actually reached target S0. If you use a bufferable write, the response for that write transaction can be returned by any intermediate component (such as a write buffer) according to the AXI protocol, so it might not even reach an exclusive access monitor before a non-exclusive response is returned.

The AXI protocol states that you must use AxCACHE encodings that ensure the target monitoring the transactions will actually see the transactions. The obvious concern here would be if the transaction was marked as cacheable, and so was stored in a cache and not accessing the target, so not monitored. But bufferable is also then a concern, and a bufferable transaction could be held in a buffer before the exclusive monitor logic, and so the exclusivity of the access is not tested when a write response is returned to the transaction source.

So yes, non-bufferable transfers might not return as quick a BRESP as a bufferable one, but we want to ensure that it is the final destination S0 target that returns the response, so a small added latency is the penalty you might have to accept. Hopefully exclusive accesses are not a significant number of your transactions, so it shouldn't be a significant performance degradation.

Hi Jzbiciak,

thank you for your comment,  for the example of interleaved accesses to  semaphore by the two CPUs, are there any specification or tutorial ?

Thank you