Swapchain errors and device lost when submitting secondary command buffer to render pass

(Edited)

Hi Aaron, 

Support for inheritedQueries is an optional feature in the API. Applications must query that the feature is supported by using vkGetPhysicalDeviceFeatures() before trying to use it. 

It should be available on newer devices (needs Mali-G710 or newer hardware, with r49p1 or newer drivers). Can you confirm whether the feature is showing as available on your Pixel, and the driver version? It should be supported on the Pixel 7, but only with the very latest software update which includes the r50p0 drivers.

Cheers, 
Pete 

(Edited)

Hi Aaron, 

Support for inheritedQueries is an optional feature in the API. Applications must query that the feature is supported by using vkGetPhysicalDeviceFeatures() before trying to use it. 

It should be available on newer devices (needs Mali-G710 or newer hardware, with r49p1 or newer drivers). Can you confirm whether the feature is showing as available on your Pixel, and the driver version? It should be supported on the Pixel 7, but only with the very latest software update which includes the r50p0 drivers.

Cheers, 
Pete 

I am querying this feature using vkGetPhysicalDeviceFeatures, and inheritedQueries is being set to 1.

According to the Vulkan device info, the driver version is 0xC800000. I am on Android 15, build number AP4A.250105.002, and shows as having no updates available. (last updated Jan 5)  I'm not sure how to check for a human-readable driver version like you're reporting.

I'm not sure how to check for a human-readable driver version like you're reporting.

Use vkGetPhysicalDeviceProperties and dump the deviceName and driverVersion fields:

    std::string name { deviceProperties.deviceName };

    uint32_t driverVersion = deviceProperties.driverVersion;
    uint32_t major = VK_VERSION_MAJOR(driverVersion);
    uint32_t minor = VK_VERSION_MINOR(driverVersion);
    uint32_t patch = VK_VERSION_PATCH(driverVersion);

The Arm standard numbering scheme is r<major>p<minor>. In your case 0xC800000 = r50p0, so this should work.

Based on this behavior, I suspect it is dropping the primary command buffer that the secondary command buffer is executed during the queue submit.

I don't quite understand what the "it" is in this case. Khronos validation layer is run before the driver sees each API call, so if the validation layer is not seeing a primary command buffer I don't see how the driver is causing it. The driver hasn't run at the point that the validation layer reads the submit data from the application. Are you sure the engine is passing in the correct command buffers on Mali?

(Although I would have expected another validation error if you just tried to submit a secondary directly)

Possibly, if timing is different on Mali and you are missing a sync, are you passing in a command buffer that has not been recorded or has been reset?

Peter Harris said:

I don't quite understand what the "it" is in this case.

I am referring to Vulkan with the queue submission.

Peter Harris said:

Khronos validation layer is run before the driver sees each API call, so if the validation layer is not seeing a primary command buffer I don't see how the driver is causing it.

This is based on my observations, though I admit that I only have limited knowledge of the inner workings of the validation layers. I know that they keep track of the commands on the command buffer in a layer above the device, but is there any communication of the driver for situations such as queue submissions? At least in this case, when the validation error fires it does so inside the call to vkQueuePresentKHR(), where the queue submission was done earlier. So if the command buffer was never "consumed" from the queue submission, I could see the validation layer may see that it wasn't executed before the present.

Peter Harris said:

Are you sure the engine is passing in the correct command buffers on Mali?

I am as sure as I possibly can be. I used printouts to verify that the correct command buffers were being written to, as well as submitted for the queue submission. Both the usage of the swapchain image at the start of the submission and the present after the submission are protected by semaphores, and I verified that the correct semaphore IDs were used for these operations as well.

Beyond that, I was able to capture the frame trace in RenderDoc, verifying that all the command buffers and related commands were submitted correctly. Additionally, the RenderDoc process that replays the commands on the device will crash with device lost when occlusion query inheritance is enabled in the secondary command buffer, and runs properly with occlusion query inheritance disabled. This is a single frame's commands in isolation in a completely separate process, which I expect would rule out many of the potential errors I could have, especially when it comes to synchronization. The flags set on the secondary command buffer inheritance info is the only difference between both runs.

Peter Harris said:

Possibly, if timing is different on Mali and you are missing a sync, are you passing in a command buffer that has not been recorded or has been reset?

The primary command buffer is always constructed on the same thread that submits it, and the threads that create the secondary command buffers are waited on before submitting them to the primary command buffer. In fact, for the TestScene tester since it only creates a single secondary command buffer, the multithreaded rendering system sees there's only one task and executes it on the main thread, so in this case there's actually no threading going on.

For the Vulkan synchronization, I have two ringbuffers of semaphores: one for the queue submissions (6, assuming 3 frames before we synchronize with the GPU and expecting up to 2 submissions per frame), and one for the images used in the swapchains (based on the number of swapchain images). The queue submission semaphore will be used to synchronize the call to vkQueuePresentKHR() to ensure the present is done after the commands finish, while the swapchain image semaphores are used to synchronize the call to vkQueueSubmit() is done once the swapchain image is ready.

When the bug exhibits itself, I see the validation errors on the second frame for the TestScene tester. It hasn't had a chance to re-use any command buffer pools, swapchain images, or semaphores before that point. The device lost error occurs the first time it attempts to synchronize with the GPU after the third frame, which is also before any of these resources have a chance to be re-used.

Thanks for double checking - I'll ask the driver team to take a look.

Yeah, it does indeed look like we have a bug in the driver which causes this to fail.

It triggers if you set VkCommandBufferInheritanceInfo.occlusionQueryEnable = VK_TRUE for a secondary command buffer and use it when there is not an active query in the primary. The workaround is to only set occlusionQueryEnable when you have an active query in the primary.

The validation error you are getting probably happens after this occurs, and is a side-effect of this earlier error. 

Sorry for the inconvenience, and thank you for reporting it. 

Cheers, 
P