• State Accepted Answer
  • Locked Locked
  • Replies 11 replies
  • Answers 1 answer
  • Subscribers 139 subscribers
  • Views 4521 views
  • Users 0 members are here
2025 survey

We are running a survey to help us improve the experience for all of our members. If you see the survey appear, please take the time to tell us about your experience if you can.

Options
Related
How was your experience today?
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Swapchain errors and device lost when submitting secondary command buffer to render pass

Aaron Barany

When testing my applications on Android 15 with a Mali-G710 MP7 (Pixel 7), I noticed that any time I submit a secondary command buffer to a render pass I get the following validation error when presenting the swap chain:

VUID-VkPresentInfoKHR-pImageIndices-01430: Validation Error: [ VUID-VkPresentInfoKHR-pImageIndices-01430 ] Object 0: handle = 0xb400007be5014a90, type = VK_OBJECT_TYPE_QUEUE; | MessageID = 0x48ad24c6 | vkQueuePresentKHR(): pPresentInfo->pSwapchains[0] images passed to present must be in layout VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR but is in VK_IMAGE_LAYOUT_UNDEFINED.
The Vulkan spec states: Each element of pImageIndices must be the index of a presentable image acquired from the swapchain specified by the corresponding element of the pSwapchains array, and the presented image subresource must be in the VK_IMAGE_LAYOUT_PRESENT_SRC_KHR or VK_IMAGE_LAYOUT_SHARED_PRESENT_KHR layout at the time the operation is executed on a VkDevice (docs.vulkan.org/.../wsi.html

After a few frames, the device is lost. I haven't encountered either issue on any other platform when running the same tester going through the same Vulkan code paths.

Here is a screenshot from RenderDoc when running the same tester on Linux with the same configuration to draw with a secondary command buffer:

I have highlighted the barriers that perform the layout transitions. Here are the contents of the barriers themselves:

Some other things to note:

  • When I start the subpass, I do provide VK_SUBPASS_CONTENTS_SECONDARY_COMMAND_BUFFERS.
  • The secondary command buffer has both ONE_TIME_SUBMIT and RENDER_PASS_CONTINUE bits set.
  • The inheritance info is set with the render pass, subpass index, and framebuffer.
  • The render pass is a single subpass. I have tried with and without multisample resolve with no difference between the two.
  • The swap buffers is waiting on the semaphore for the submit.
  • I wait on fences before re-using any resources from a few frames ago. For command buffers, I reset the command buffer pool once it is no longer in use.

When debugging, I even tried waiting for the GPU to be idle before processing the submit, and I still had the same behavior. The following can avoid both the validation and device lost issues:

  • Put drawing commands directly to the main command buffer rather than using a secondary command buffer.
  • Omitting the submission of the secondary command buffer to the primary command buffer.
  • Removing all draw calls from the secondary command buffer.

So far I haven't been able to find the source of the error for the image transitions, command buffer setup, or queue submission and swapchain presentation. I see that Mali-G710 was advertised as the first Mali GPU with native secondary command buffer support. Are there extra steps that I'm missing when the hardware support is used for Mali, such as extra barriers? Could this be a driver issue, perhaps android-specific with the swapchains?

The project is open source, I can share it and instructions to reproduce if need be.

Parents
  • 0 in reply to Peter Harris
    Peter Harris said:
    I don't quite understand what the "it" is in this case.

    I am referring to Vulkan with the queue submission.

    Peter Harris said:
    Khronos validation layer is run before the driver sees each API call, so if the validation layer is not seeing a primary command buffer I don't see how the driver is causing it.

    This is based on my observations, though I admit that I only have limited knowledge of the inner workings of the validation layers. I know that they keep track of the commands on the command buffer in a layer above the device, but is there any communication of the driver for situations such as queue submissions? At least in this case, when the validation error fires it does so inside the call to vkQueuePresentKHR(), where the queue submission was done earlier. So if the command buffer was never "consumed" from the queue submission, I could see the validation layer may see that it wasn't executed before the present.

    Peter Harris said:
    Are you sure the engine is passing in the correct command buffers on Mali?

    I am as sure as I possibly can be. I used printouts to verify that the correct command buffers were being written to, as well as submitted for the queue submission. Both the usage of the swapchain image at the start of the submission and the present after the submission are protected by semaphores, and I verified that the correct semaphore IDs were used for these operations as well.

    Beyond that, I was able to capture the frame trace in RenderDoc, verifying that all the command buffers and related commands were submitted correctly. Additionally, the RenderDoc process that replays the commands on the device will crash with device lost when occlusion query inheritance is enabled in the secondary command buffer, and runs properly with occlusion query inheritance disabled. This is a single frame's commands in isolation in a completely separate process, which I expect would rule out many of the potential errors I could have, especially when it comes to synchronization. The flags set on the secondary command buffer inheritance info is the only difference between both runs.

    Peter Harris said:
    Possibly, if timing is different on Mali and you are missing a sync, are you passing in a command buffer that has not been recorded or has been reset?

    The primary command buffer is always constructed on the same thread that submits it, and the threads that create the secondary command buffers are waited on before submitting them to the primary command buffer. In fact, for the TestScene tester since it only creates a single secondary command buffer, the multithreaded rendering system sees there's only one task and executes it on the main thread, so in this case there's actually no threading going on.

    For the Vulkan synchronization, I have two ringbuffers of semaphores: one for the queue submissions (6, assuming 3 frames before we synchronize with the GPU and expecting up to 2 submissions per frame), and one for the images used in the swapchains (based on the number of swapchain images). The queue submission semaphore will be used to synchronize the call to vkQueuePresentKHR() to ensure the present is done after the commands finish, while the swapchain image semaphores are used to synchronize the call to vkQueueSubmit() is done once the swapchain image is ready.

    When the bug exhibits itself, I see the validation errors on the second frame for the TestScene tester. It hasn't had a chance to re-use any command buffer pools, swapchain images, or semaphores before that point. The device lost error occurs the first time it attempts to synchronize with the GPU after the third frame, which is also before any of these resources have a chance to be re-used.

Reply
  • 0 in reply to Peter Harris
    Peter Harris said:
    I don't quite understand what the "it" is in this case.

    I am referring to Vulkan with the queue submission.

    Peter Harris said:
    Khronos validation layer is run before the driver sees each API call, so if the validation layer is not seeing a primary command buffer I don't see how the driver is causing it.

    This is based on my observations, though I admit that I only have limited knowledge of the inner workings of the validation layers. I know that they keep track of the commands on the command buffer in a layer above the device, but is there any communication of the driver for situations such as queue submissions? At least in this case, when the validation error fires it does so inside the call to vkQueuePresentKHR(), where the queue submission was done earlier. So if the command buffer was never "consumed" from the queue submission, I could see the validation layer may see that it wasn't executed before the present.

    Peter Harris said:
    Are you sure the engine is passing in the correct command buffers on Mali?

    I am as sure as I possibly can be. I used printouts to verify that the correct command buffers were being written to, as well as submitted for the queue submission. Both the usage of the swapchain image at the start of the submission and the present after the submission are protected by semaphores, and I verified that the correct semaphore IDs were used for these operations as well.

    Beyond that, I was able to capture the frame trace in RenderDoc, verifying that all the command buffers and related commands were submitted correctly. Additionally, the RenderDoc process that replays the commands on the device will crash with device lost when occlusion query inheritance is enabled in the secondary command buffer, and runs properly with occlusion query inheritance disabled. This is a single frame's commands in isolation in a completely separate process, which I expect would rule out many of the potential errors I could have, especially when it comes to synchronization. The flags set on the secondary command buffer inheritance info is the only difference between both runs.

    Peter Harris said:
    Possibly, if timing is different on Mali and you are missing a sync, are you passing in a command buffer that has not been recorded or has been reset?

    The primary command buffer is always constructed on the same thread that submits it, and the threads that create the secondary command buffers are waited on before submitting them to the primary command buffer. In fact, for the TestScene tester since it only creates a single secondary command buffer, the multithreaded rendering system sees there's only one task and executes it on the main thread, so in this case there's actually no threading going on.

    For the Vulkan synchronization, I have two ringbuffers of semaphores: one for the queue submissions (6, assuming 3 frames before we synchronize with the GPU and expecting up to 2 submissions per frame), and one for the images used in the swapchains (based on the number of swapchain images). The queue submission semaphore will be used to synchronize the call to vkQueuePresentKHR() to ensure the present is done after the commands finish, while the swapchain image semaphores are used to synchronize the call to vkQueueSubmit() is done once the swapchain image is ready.

    When the bug exhibits itself, I see the validation errors on the second frame for the TestScene tester. It hasn't had a chance to re-use any command buffer pools, swapchain images, or semaphores before that point. The device lost error occurs the first time it attempts to synchronize with the GPU after the third frame, which is also before any of these resources have a chance to be re-used.

Children
  • 0 Arm Employee Badge in reply to Aaron Barany

    Thanks for double checking - I'll ask the driver team to take a look.

  • +1 Arm Employee Badge in reply to Peter Harris

    Yeah, it does indeed look like we have a bug in the driver which causes this to fail.

    It triggers if you set VkCommandBufferInheritanceInfo.occlusionQueryEnable = VK_TRUE for a secondary command buffer and use it when there is not an active query in the primary. T    he workaround is to only set occlusionQueryEnable when you have an active query in the primary.

    The validation error you are getting probably happens after this occurs, and is a side-effect of this earlier error. 


    Sorry for the inconvenience, and thank you for reporting it. 

    Cheers, 
    P