Hello,
I'm contribute to a bare-metal project (small operating system kernel inclusive user space). The project worked without issues in QEMU. But every time failed on real hardware because a dead lock which was only on real hardware. The dead lock was caused by the Spinloop and Mutex implementation which are using the exclusive load (ldaxr, ldxr, ...) and store (stxr, ...) instructions. These instructions doesn't work correctly when the caches are disabled.
Can somebody explain the reason of this behavior? Unfortunately I didn't found any hint in the architecture reference manual.
Mostly likely explanation is that the memory being targeted doesn't support exclusive accesses.
Exclusives work by tagging the accessed address (well, a block containing the address). The exclusive load sets the "tag", the exclusive store is conditional on the tag still being there. The "tag" is stored in an Exclusive Monitor. Generally, you will have Local Monitors and Global Monitors. The Local Monitor only have visiblity of that core's accesses, while the Global Monitor has visibility of other agents' accesses as well. Which monitor is used for a given load/store exclusive depends on the attributes of the accessed address.
When the address was cache-able, you were using the Local Monitor - which is part of the core. When you made it non-cacheable, the processor switched to relying on the Global Monitor - which is part of the memory system. Which addresses have Global Monitor support is SoC specific (for example, an SoC might support it for SRAM but not DRAM). So you need to make sure that exclusives which are non-cacheable are targeting addresses that the SoC supports exclusives to.
My explanation is simplified, for a more verbose description try this: https://support.arm.com/documentation/110478/0100/Exclusives