This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

High Load/Store Cycles

Hi, i'm using performance advisor together with streamline to profile our game.
The report says our GPU seems to be busy with load/store operations. The optimization advice is mainly about how to optimize compute shaders.
The GPU cycles indicates we are fragment bound. (Total 20K Fragment 18K Non-Fragment 16K)
The GPU external read bandwidth is 137M, write bandwidth is around 350M

Our game is quite complex and the only compute shader related pass is some postprocess related to tone-mapping.
So i looked into streamline's related performance counters:
Streamline sample duration is 100ms, total load store cycles is 3526K, and it's about 1.8 times more than other ALU operations.

Load/store total issues:10.19 mega-cycles
Load/store full read issues:0.31 mega-cycles
Load/store partial read issues:3.84 mega-cycles
Load/store full write issues:4 mega-cycles
Load/store partial write issues:2.03 mega-cycles
Load/store atomic issues:< 0.01 mega-cycles

We're testing on a G77MP16 GPU, docs says it's LSC bandwidth is designed to be 32 Byte / Cycle

Load/store bytes read from L2 per access cycle:8 byte
Load/store bytes read from external memory per access cycle:3 byte
Load/store bytes written to L2 per access cycle:16 byte

Load/store read bytes from L2 cache:32.48 MB
Texture read bytes from L2 cache:8.48 MB
Load/store read bytes from external memory:12.3 MB
Texture read bytes from external memory:6.08 MB
Load/store write bytes:97 MB
Tile buffer write bytes:2.56MB

Full quad warp rate:96%
Diverged instruction issue rate:<1%
All registers warp rate:25%
Constant tile kill rate:7%

The Load/Store write bytes seems to consume most L2 cache's bandwidth

Although above counters doesn't indicate any problem related to external memory load/store, but theses counter looks suspicious:

Output external read bytes:409 MB / 100ms,137 MB / Frame
Output external write bytes:1037 MB / 100ms,342 MB / Frame

Output external read stall rate:6%
Output external write stall rate:200%

The external write stall rate is 200% ...

So, to my knowleage, performance advisor says we have a load/store problem. And there's at least 2 types of load/stores.
One for L2 cache, one for external memory.

For L2 cache load/stores, we're write bound? But when does GPU write to L2 cache? Like when register spill happens(We have a 25% of all register wrap rate)? Like when filling L2 cache with geometry / thread stack data for shader wraps? And i tried to show/hide some portion of our game, the execution engine & load/store cycles goes down together. What should i do? I'm a little confused here, what could cause a high L2 load/store write bandwidth.

For external memory load/store, the counters says we're still write bound. I understand writing to external memory is time consuming.
Does 14GB/s is too much for the external memory system? Should we try to optimize external write bandwidth as well?

Parents
  • The max number of work register is 64. if, for example, the position or varying variant exceeds this limit, what would happen?

    If you exceed the 64 register capacity a shader would start to spill to stack. You can see that's not happening here - stack spilling is reporting as false. The compiler tries hard to not to do this - spilling in a GPU is very expensive in terms of bandwidth due to the thread count involved. 

    BTW, what's the relation ship between LSC(16K) and load/store L2 Cache, do they mean the same thing?

    LSC - level 1 cache in the shader core, only used for load/store data, and typically 16KB in size.

    L2 - level 2 unified cache, shared by multiple shader cores, used for most types of data (shaders, descriptors, buffers, textures, etc), and typically anywhere up to 2-4MB for large GPU designs. Slower to access than the L1 though. 

    And when exceeding uniform register limit(16 i guess?), will mali-gpu do the same thing to load/store uniform registers in LSC?

    Uniform register limit is large (128 registers per draw), so you have to try very hard to exceed it. If you do, or if uniform registers can't be used (e.g. dynamic array index), then yes will load from load/store cache. 

    If it's true, does this mean some part of our load/store pressure comes from using too much registers?

    Without seeing the shader, it's impossible for me to give a definitive answer. You're not spilling, so work register pressure isn't the problem, but you might be loading uniforms from memory (either due to size or how they are being used).

    For vertex shaders, loading vertex attributes is more likely to be the major cost, as that goes down the load/store path.

Reply
  • The max number of work register is 64. if, for example, the position or varying variant exceeds this limit, what would happen?

    If you exceed the 64 register capacity a shader would start to spill to stack. You can see that's not happening here - stack spilling is reporting as false. The compiler tries hard to not to do this - spilling in a GPU is very expensive in terms of bandwidth due to the thread count involved. 

    BTW, what's the relation ship between LSC(16K) and load/store L2 Cache, do they mean the same thing?

    LSC - level 1 cache in the shader core, only used for load/store data, and typically 16KB in size.

    L2 - level 2 unified cache, shared by multiple shader cores, used for most types of data (shaders, descriptors, buffers, textures, etc), and typically anywhere up to 2-4MB for large GPU designs. Slower to access than the L1 though. 

    And when exceeding uniform register limit(16 i guess?), will mali-gpu do the same thing to load/store uniform registers in LSC?

    Uniform register limit is large (128 registers per draw), so you have to try very hard to exceed it. If you do, or if uniform registers can't be used (e.g. dynamic array index), then yes will load from load/store cache. 

    If it's true, does this mean some part of our load/store pressure comes from using too much registers?

    Without seeing the shader, it's impossible for me to give a definitive answer. You're not spilling, so work register pressure isn't the problem, but you might be loading uniforms from memory (either due to size or how they are being used).

    For vertex shaders, loading vertex attributes is more likely to be the major cost, as that goes down the load/store path.

Children