This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

High Load/Store Cycles

Hi, i'm using performance advisor together with streamline to profile our game.
The report says our GPU seems to be busy with load/store operations. The optimization advice is mainly about how to optimize compute shaders.
The GPU cycles indicates we are fragment bound. (Total 20K Fragment 18K Non-Fragment 16K)
The GPU external read bandwidth is 137M, write bandwidth is around 350M

Our game is quite complex and the only compute shader related pass is some postprocess related to tone-mapping.
So i looked into streamline's related performance counters:
Streamline sample duration is 100ms, total load store cycles is 3526K, and it's about 1.8 times more than other ALU operations.

Load/store total issues:10.19 mega-cycles
Load/store full read issues:0.31 mega-cycles
Load/store partial read issues:3.84 mega-cycles
Load/store full write issues:4 mega-cycles
Load/store partial write issues:2.03 mega-cycles
Load/store atomic issues:< 0.01 mega-cycles

We're testing on a G77MP16 GPU, docs says it's LSC bandwidth is designed to be 32 Byte / Cycle

Load/store bytes read from L2 per access cycle:8 byte
Load/store bytes read from external memory per access cycle:3 byte
Load/store bytes written to L2 per access cycle:16 byte

Load/store read bytes from L2 cache:32.48 MB
Texture read bytes from L2 cache:8.48 MB
Load/store read bytes from external memory:12.3 MB
Texture read bytes from external memory:6.08 MB
Load/store write bytes:97 MB
Tile buffer write bytes:2.56MB

Full quad warp rate:96%
Diverged instruction issue rate:<1%
All registers warp rate:25%
Constant tile kill rate:7%

The Load/Store write bytes seems to consume most L2 cache's bandwidth

Although above counters doesn't indicate any problem related to external memory load/store, but theses counter looks suspicious:

Output external read bytes:409 MB / 100ms,137 MB / Frame
Output external write bytes:1037 MB / 100ms,342 MB / Frame

Output external read stall rate:6%
Output external write stall rate:200%

The external write stall rate is 200% ...

So, to my knowleage, performance advisor says we have a load/store problem. And there's at least 2 types of load/stores.
One for L2 cache, one for external memory.

For L2 cache load/stores, we're write bound? But when does GPU write to L2 cache? Like when register spill happens(We have a 25% of all register wrap rate)? Like when filling L2 cache with geometry / thread stack data for shader wraps? And i tried to show/hide some portion of our game, the execution engine & load/store cycles goes down together. What should i do? I'm a little confused here, what could cause a high L2 load/store write bandwidth.

For external memory load/store, the counters says we're still write bound. I understand writing to external memory is time consuming.
Does 14GB/s is too much for the external memory system? Should we try to optimize external write bandwidth as well?

Parents
  • This chart shows Shader Core's different unit's performance in cycle count, right?

    Yes, it shows the utilization of the major functional part of each pipeline - i.e. the bit that does real work.  

    It is deliberately simplified and can't show all the detail of every pipeline, so its a simplification and there may be bottlenecks in each pipe which don't show up here if content start stressing the pipe in weird ways. Streamline has the full counter set and can show more detail.

    So the high Load/Store cycles is caused by load/store data from unified L2 Cache and external DRAM, which don't include LSC bandwidth, is this correct?

    The LSC counter just shows L1 cache accesses. Each access might not access L2 or external memory at all if the data is already in the L1. 

    If we confirm we consumes too much external memory write bandwidth, what's the main cause of writing to external DRAM?

    Intermediate outputs from the vertex shader are stored back to memory before being read by the fragment shader, so the main cause of load/store writes is vertex shader outputs. Minimizing vertex count and reducing precision (use mediump outputs from the vertex shader as much as possible - usually only need highp for position and depth) can help to reduce the write bandwidth here.

    Other options are writes from compute shaders (either to buffers or to images).

Reply
  • This chart shows Shader Core's different unit's performance in cycle count, right?

    Yes, it shows the utilization of the major functional part of each pipeline - i.e. the bit that does real work.  

    It is deliberately simplified and can't show all the detail of every pipeline, so its a simplification and there may be bottlenecks in each pipe which don't show up here if content start stressing the pipe in weird ways. Streamline has the full counter set and can show more detail.

    So the high Load/Store cycles is caused by load/store data from unified L2 Cache and external DRAM, which don't include LSC bandwidth, is this correct?

    The LSC counter just shows L1 cache accesses. Each access might not access L2 or external memory at all if the data is already in the L1. 

    If we confirm we consumes too much external memory write bandwidth, what's the main cause of writing to external DRAM?

    Intermediate outputs from the vertex shader are stored back to memory before being read by the fragment shader, so the main cause of load/store writes is vertex shader outputs. Minimizing vertex count and reducing precision (use mediump outputs from the vertex shader as much as possible - usually only need highp for position and depth) can help to reduce the write bandwidth here.

    Other options are writes from compute shaders (either to buffers or to images).

Children