How to correctly implement/interpret xrReleaseSwapchainImage specification?

[risa2000]

From the current OpenXR spec (https://registry.khronos.org/OpenXR/specs/1.1-khr/html/xrspec.html), bold face is mine:

xrReleaseSwapchainImage (excerpt):

• Once an application is done submitting commands that reference the swapchain image, the application must release the swapchain image.
• xrEndFrame will use the most recently released swapchain image. In each frame submitted to the compositor, only one image index from each swapchain will be used.

From xrReleaseSwapchainImage spec. it is not clear whether the client calls xrReleaseSwapchainImage right after submitting all commands to the command queue (but possibly not having the image rendered by GPU yet), or client must synchronize on GPU work before calling the function.

1. If it is a client responsibility then the rest is fine
2. If however it is a runtime responsibility then the most recently released swapchain image may not be ready for the compositor (yet) at the time it gets to it by following xEndFrame call, so it may need to
   • use the "old" image still for compositing the frame - which could introduce all sorts of different inconsistencies in what is composited.
   • reproject the "old state" from the previous xEndFrame call, but this may unnecessary block some images, which may have been already replaced by newer ones by the client, and lose the images which has been finished.

Having the compositor waiting for the image to be released by GPU may lead to lost vsync.

So the common sense would suggest that the synchronization would be the responsibility of the client app, but after having look at hello_xr demo I did not find any such mechanism. So what is the right approach for the runtime?

Another unclarity is related to xrEndFrame and the possibility for client to not call xrReleaseSwapchainImage before.

From OpenXR spec. - Frame Submission (excerpt):
An application may call xrEndFrame without having called xrReleaseSwapchainImage since the previous call to xrEndFrame for any swapchain passed to xrEndFrame.

What is the right approach for the runtime then?

1. Should it implicitly call xrReleaseSwapchainImage on each image which has been "waited" on?
2. Should it ignore the image from the views?
3. Should it take the most recently released swapchain image (i.e. some older image from the swapchain, which was released already)?

[rpavlik]

You call xrReleaseSwapchainImage once you are done submitting your draw commands, the runtime will do something like inserting a fence and checking asynchronously to see if it is done. (I think this is your option 2, in your first list. The runtime should not block on the image you submitted: if it's not ready in time, it will re-use your old one.)

If you call xrEndFrame without calling xrReleaseSwapchainImage during that "frame", the runtime will reproject the frame from the last time that you called xrReleaseSwapchainImage. Usually you do not want to do this, but sometimes you might have a layer that is only infrequently updated. (I think this is your option 3 in your second list.)

Does this help?

[risa2000]

@rpavlik
Thanks, you answered my questions! I have just two comments.

ad 1) If the runtime/compositor should not block on (possibly still being rendered) "last released" image (which makes sense), taking an older released image may lead to situation where the compositor is compositing a mix of "current" and "old" released/finished images.

To elaborate - until now I understood the spec. in a way that the app defines (implicitly) a set of images which should be composited together into a particular frame, by calling xrEndFrame and the images are exactly those "last released" in respective swapchains referenced in XrFrameEndInfo.
Using your interpretation, the images which will be composited are not necessarily those "last released", but those "last finished", which is a subtle but important difference as this way the client cannot be sure what actually ends up in the composited frame. It may not be what was released immediately prior to xrEndFrame call.

ad 2) I guess I read "not calling xrReleaseSwapchainImage" as client missing it as the last part of the whole sequence (Acquire, Wait, and Release). While your interpretation suggests that the meaning is rather the client not rendering a new image in a specific swapchain at all, but still using the swapchain for composition (in xrEndFrame). In that case the compositor is still using "the last released/finished" image, it just might be one which is not updated regularly. Makes sense.

[risa2000]

After some thought, I suggest that the specification makes the distinction between "last released" and "last finished" image and use the right one in the swapchain management context (going with the wording/explanation along your answer).

I also suggest clarifying the "not calling xrReleaseSwapchainImage" with an explanation you gave above.

Otherwise this topic can be closed.

[rpavlik-bot]

An issue (number 2387) has been filed to correspond to this issue in the internal Khronos GitLab (Khronos members only: KHR:openxr/openxr#2387 ), to facilitate working group processes.

This GitHub issue will continue to be the main site of discussion.