A53 latencies / jitter (memory access)?

Do calculation in bigger memory arrays in a quad core A53 SoC, using all 4 cores and observe a huge jitter in execution times. That is even the case if data are always same and at same location - so really repeatable situation. In an example the range for operation is between 90...110ms.   

I am aware that my OS (Linux, test different kernels up to 6.8) isn't an RT OS. (RT patches make things even worse here). But without other load of usual "housekeeping" I would not expect differences of several ms or even more than 20ms in doing calculation . Also I don't see that  behaviour in a Dual Cortex A9 or running that task on A53 disabling 3 cores - which is no real option from performance point of view. 

So I suspect that very little and short interrupts or scheduler events (that can't be avoided) can have  massive impact on cache or memory access. In kernelshark I can see that cores really go to idle for some ms instead of continue operations. To  make it clear - there are no dependencies in that operation - so each core can run its parts. I even tried to isolate calculation cores from "normal" system. That improves situation a bit. But as soon as there is additional (even light) load on that system core, the jitter of the processing in the 3 other cores increase again. 

Is it a known issue on A53 and is there something I can do (without completely changing software, which isn't possible) to get less jitter here?