We've recently encountered a performance problem in our unity game and I suspect that the reason was the CPU spikes in mali-cmar-backend thread.
As you can see from UnityMain thread and GpuCompeletion, there is no CPU Bound or GPU Bound in this test scene. The camera position is fixed in this test, and the GPU pressure should be the same in the whole systrace.
So the only clue seems to be this spike in the mali-cmar-backend thread. I tried googling mali-cmar-backend, but very little results was found, and they are not very helpful. So I wonder if I can have some information here, mainly about these two question:
1. What is this mali-cmar-backend thread? Is this CMAR the abbreviation of Control Memory Address Register?
2. What may cause this strange spikes in this thread?
Sorry about the broad questions, any ideas or keywords for me to dig into will be very helpful. Thanks in advance!
Our game is using a ton of uniform/UBO (likely far over 8k), which had caused some rendering problem on other devices eariler. Not sure if this is related to this issue.
We can reproduce this problem on a Exynos 2100 (Mali-G78) device and a kirin 820 (Mali-G77) device, but not all Mali GPUs have this problem.
We are using OpenGL ES 3.2.
Hi, this sounds a new and interesting problem... What driver version is it using on the 2 devices you're getting it on?
If possible, could you please send a reproducer (ideally cut down to just the issue, although this looks like that might be tricky?) to developer at arm dot com? We'd like to profile what's going on.
Thank you for your fast reply!
The driver version for exynos 2100 is `GLES: ARM, Mali-G78, OpenGL ES 3.2 v1.r32p1-01bet2-mbs2v39_0.f99ba4208fd3cd75b7876c6eee932fcf`
and for kirin chip is `GLES: ARM, Mali-G77, OpenGL ES 3.2 v1.r21p0-01rel0.4af6038a9bcc0e6ebbaecc1c676406b0`
I'm sorry that we cannot provide the apk now, because the game is not released yet and the development server is LAN access only. It is possible to profile the release package when the game is online (ideally in a month or two).
Instead, we can provide some systrace file and GPU counter data, if they are helpful.
First guess from the team was that there could be an excessive amount of flushes (but cause unknown). Thought I'd mention in case that gives you ideas (too much incremental rendering? or...)
systrace and GPU counters would be good thanks, in lieu of an apk. Please send to developer at arm dot com email and mention this thread.
I've just sent some systrace files and counter data.
I think we are not using too many flushes (if you mean glFlush or glFinish). We didn't manually call them, and I can't find any in the renderdoc captures. We are using some glflushmappedbufferrange (at most 15 per frame). I'm not sure if that is a type of flush, and I'll try to disable them and test again.
Though mentioned in the email, I'd also want to repeat it here, so that other readers won't miss some detail.
When collecting GPU data, I found out that we cannot reproduce the issue as long as SystemProfiler (from Huawei) or Streamline is connected to the game and actived. Take SystemProfiler as an example. we first reproduce this issue and confirm it is happening by seeing a spike in systrace. Then we click the start button to collect GPU counters, and the fps just went back to normal (also, the spike in mali-cmar-backend thread disappears). Then we click stop button to stop collecting data, the same issue just immediately happens again (we can see the spikes again in systrace).
May I ask if glInvalidateFrameBuffer or glBindFrameBuffer would implicitly call a flush in the drivers? We do switch framebuffers many times in a frame.