Add sample for HDR display formats

[SaschaWillems]

Most modern games support HDR displays, and Vulkan supports this with respective surface formats and color spaces. E.g. VK_FORMAT_A2B10G10R10_UNORM_PACK32 and VK_COLOR_SPACE_HDR10_ST2084_EXT.

We should add a true HDR sample to our repo, that not only uses HDR formats for internal calculations (like the HDR sample), but also outputs HDR.

[alecazam]

Nvidia also has rgba16f + extended srgb linear color space. This is ideally what apps could use, since using Rec2020 is hard where UI is Rec709 color space, and the render and final framebuffer is Rec2020_PQ.

But there seem to be no good examples of handling dpi, dpi window scaling, hdr, and fullscreen exclusive all needed to make a game on Windows. Many of the discussions I found also mention using DXGI and it's "flip model" for the accelerated borderless window. Nvidia has a setting for that now, but AMD/Intel don't. So then presents go through GDI copies.

[jpark37]

It would be great if you could demonstrate how to get HDR working on Android. I was not able to figure it out myself.

I was able to get Vulkan HDR working on Windows, but it doesn't seem to play as nicely as DXGI. I have a multi-monitor setup where both displays are HDR capable, but I have one of them set to SDR in the OS, and Vulkan HDR forces that display into HDR mode against my wishes. I would not recommend using Vulkan for HDR output on Windows at this time.

Nvidia also has rgba16f + extended srgb linear color space. This is ideally what apps could use, since using Rec2020 is hard where UI is Rec709 color space, and the render and final framebuffer is Rec2020_PQ.

This method (tested via D3D/DXGI) was slower for both fullscreen and windowed scenarios on my NVIDIA card, presumably because of the double-sized swap chain. Speed aside, it's probably still useful for high-precision though.

[alecazam]

On Win10, you set the HDR mode for each monitor. Once enabled for a monitor, then you can set SDR content scaling. From DXGI, you can test the hdr capabilities of each monitor, and switch between 16f and 8srgb. There should really be an api in Vulkan for this, but Vulkan is trying to stay vendor neutral like OpenGL which results in a big mess at the swapchain level just like OpenGL.

Windows11 finally has a 16f DWM that may apply to multiple monitors. This is similar to what Apple has been doing for over a decade now. This allows color correction, but they also have a new compositor api that can frame pace.

[jpark37]

MS says system composition for HDR was added in Windows 10 1703. Pretty sure that path doesn't have issues with multiple monitors, but I don't remember how much Windows 10 testing I did, so I might be remembering wrong. I haven't seen any issue reports about it either though.

Windows 11 22H2 added wide-color SDR if that's what you're thinking of.

Speaking of Apple, EDR support was added to iOS last year, and I haven't figured out how to toggle EDR on/off dynamically (using Metal directly). I was messing with Unreal, and EDR only seems to work if I enable it near launch, and not where it normally (re-)initializes the swap chain. It would be nice if this sample could demonstrate HDR practices on Mac/iOS using MoltenVK if that's a thing.

[alecazam]

MS has had scRGB in the DWM since 1703 (3/17), then they handle all hdr conversion to Rec2020/2100 after that. Win11 is the same system, but now formalized with fast flip support.

Seems that Vulkan never added an scRGB format though. So you can store 16f in scRGB, but not get it out to the swap chain. The AMD HDR sample is using an AMD extension and their AMD-specific api to get scRGB. And the games seem to all be bypassing the Vulkan swapchain entirely, and grafting on DXGI with a shared surface. This is so much work, when Vulkan should just already have scRGB color space. I shouldn't have to wait on Vulkan 1.5 for something as basic as this.

So Vulkan basically forces use of ST2084 (Rec2020/2100) which has numerous problems for blitting/blending sRGB UI + renders. Thats why extended P3 and scRGB colorspaces exist. And all Windows does then is convert it back to 16f scRGB, composting other HWND (if needed), and then sending it back out to Rec2020. I could see some benefit to Rec2020 if it could send it out directly to the display, but it's hard to bypass the DWM compositor these days.

[alecazam]

These are good references

https://forums.larian.com/ubbthreads.php?ubb=showflat&Number=680146

AMD using their own color spaces. Also seem to require fullscreen exclusive which is dead. But other articles indicate freesync supports fullscreen borderless windowed.

https://gpuopen.com/learn/using-amd-freesync-2-hdr-color-spaces/

I have no idea yet on Nvidia. The articles are ancient and before Microsoft added HDR support and involve NVAPI which I really don't want to have to access.

[mirh]

Fullscreen exclusive isn't dead as in "deleted from the OS", it's dead just as in "used less than in the past".
The new Windows fullscreen optimizations may perhaps give you the former impression by trying to upgrade D3D windows in most situations, but since microsoft won't touch vulkan or opengl with a 10 foot pole it's ironically here that FSE could almost be taken for granted universally (in fact, before VK_EXT_full_screen_exclusive made the switch explicit/documented, in order not to use it once you went fullscreen you had to bang your head for hours to find the right window styles for borderless windowed only)

With this said and premised, @Kaldaien wasn't saying that you need a DXGI swapchain to have scRGB.
He's saying that you need the interop in order for HDR to work at all.
Yes, there is still the fullscreen exclusive loophole around this limitation, but this isn't 2017 and just like you shouldn't use nvapi or ags anymore it would also seem kinda lame to give up on windowed modes (FSE may also be the reason a certain "vulkan hdr" situation could upset some multi-monitor configurations).

Then I'm not really sure how much modern AAA games set up themselves the dx surface (Id was doing this with opengl almost a decade ago), as opposed to just the drivers gracefully doing the work in-their-place behind the scenes (btw idk how AMD used to handle this in the past, but at least now they are also aboard).. anyhow, insofar as this is supposed to be an important sample I feel like it should cover the entire comprehensive procedure.

p.s. android sounds actually pretty easy, provided you have a phone supporting the right extensions
p.p.s. linux still isn't there as a platform, but if you don't rely on a compositor you may already try to sketch some code
p.p.p.s. I don't think supporting VK_FORMAT_R16G16B16A16_SFLOAT as a surface format has anything to do with amd extensions, or the vulkan specification? It's just a driver choice AFAIU.

[jpark37]

android sounds actually pretty easy, provided you have a phone supporting the right extensions

I can get the Android and Vulkan APIs to claim the right HDR things. It just refuses to actually display HDR; all extended values get clipped. A sample would be nice to see what I might be doing wrong.

[mirh]

Qualcomm also reports the same thing.. And I now found some actual official sample.
I think setup should work just like with WCG, except you need android:colorMode="hdr" in the manifest and/or call setColorMode(ActivityInfo.COLOR_MODE_HDR) instead?

[SaschaWillems]

Just a small request: This is a tracker issue for a new sample and not a general HDR discussion. Feel free to move this to https://github.com/KhronosGroup/Vulkan-Samples/discussions.

[mkarlesky]

@SaschaWillems @jpark37 Briefly, how would the existing HDR sample need to be modified to display HDR10 (on Android)? Asking for a friend…

[alecazam]

Good luck with Android. Google advertises HDR support on phones, but when you delve then you find out that Vulkan never publishes any HDR10 color spaces (f.e. ST2084). I assume that Android's system UI blending can't deal with HDR10, but video never has text above it. Of course every other system has HDR in Metal and Vulkan.

[jpark37]

you find out that Vulkan never publishes any HDR10 color spaces

I was able to enumerate VK_FORMAT_R16G16B16A16_SFLOAT/VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT from vkGetPhysicalDeviceSurfaceFormatsKHR on at least one of my devices. I may have seen VK_FORMAT_A2B10G10R10_UNORM_PACK32/VK_COLOR_SPACE_HDR10_ST2084_EXT as well; I can't remember, but extended scRGB was there for sure.

I have not gotten this to work, but this is what I've tried on my own project:

• android:colorMode="hdr" in AndroidManifest.xml and logging of isHdr() from my android.view.Display claims that HDR is supported.
• Enumerate the VkSurfaceFormatKHR as mentioned above and pass format/space to vkCreateSwapchainKHR.
• Do HDR output math in shaders.

To keep this sample-related, a sample for this would be nice.

I found this mention of "10-bit HDR gaming" being available for Snapdragon when I was trying to get this working, but maybe it's not true at all:
https://www.xda-developers.com/pubg-mobile-90fps-true-10-bit-hdr-support/

[tomadamatkinson]

Worth taking a look at @SaschaWillems GPU Info website to see what HDR device support on Android is like on the last few generations. Support for formats/extensions can be very device specific

[SaschaWillems]

There is only one single Android device that reports a ST2084/ST2086 colorspace, and that's a NVIDIA shield. From my experience everything said here about HDR on Android with Vulkan holds true. There seems to be a way to use HDR on Android (not to be confused with wide color gamut) but documentation is very sparse.

[alecazam]

Having just worked on HDR support for a game across many Vulkan and non-Vulkan platforms, I can say that HDR is quite the mess. But a sample should be do-able on Win.

On Win10, the OS and IHV control panels control aspects of HDR. The OS has an "HDR on" setting if the monitor is capable, the monitor has settings to enable HDR, and then DXGI detects 10-bit+ color and ST2084 from the monitor, and then monitor refresh rate and cabling must handle the bandwidth. I had to drop my 165Hz panel to 144Hz to avoid banding. DXGI is needed since Vulkan offers no support for swap chain information. A DXGI surface must be shared to replace the Vulkan swapchain, or else Nvidia has a fast flip setting where it creates this DXGI surface behind the scenes.

Also many "HDR" monitors really aren't. They are 8-bit panels, or have no local dimming, have low nits (200-400) or require switching the Nvidia control panel from RGB4 to YCbCr422 to still run HDR10 at 60Hz (Samsung M8). So that's less color fidelity. But televisions seem to be getting this right, and newer monitors should fix this (Tempest G27).

Windows has wanted scRGB in the past for the DWM to handle the compositing. FSE was often the only path to HDR10. But now that FSE is dead, and fullscreen borderless window is the norm to copy macOS, it's unclear what to specify. In windowed mode, HDR10 (10A2) may be converted back to scRGB 16f to do the composite and then out to the display. But most games coming from console go direct to 10A2. You also need to split off the UI into a premul-blended sRGBA8 texture and composite that in the tonemapper. And there are many subtleties in handling blending srgb + hdr10 data. There's a good Nvidia GDC talk on that.

macOS handles HDR much more simply via EDRHeadroom = maxHDRBrightness / maxSDRBrightness. On Win, there is this clunky SDR brightness slider that I don't believe apps have access to. So on Win, it's EDRHeadroom = maxHDRBrightness / 100 nits. But most monitors these days don't use 100 nits. I wish Win11 would just copy this approach. But it means that you need to tonemap HDR to the range above that.

So typical tonemapping values for EDR headroom:
M1 = 1600 nits / 400 nits = 4, 400 is dynamic, and also monitor brightness sliders affect all this
Win = 1200 nits / 100 nits = 12, monitor brightness slider affects maxHDRBrightness

[jpark37]

There is only one single Android device that reports a ST2084/ST2086 colorspace, and that's a NVIDIA shield

Looks like this might have been a mistake because the surface format was removed from newer driver releases. I see it listed for 361.0.0.0, but not past that. Saved me an unnecessary purchase. :P

I did find this Qualcomm sample recently although I'm skeptical it will work. Maybe vkSetHdrMetadataEXT is required on Android because that's one thing it's doing that I'm not. Windows didn't need it IIRC. I'll probably give it a try soon:
https://github.com/quic/adreno-gpu-vulkan-code-sample-framework/tree/main/samples/hdrSwapchain

[jpark37]

The Qualcomm sample didn't build for me out of the box and crashes on my phone on startup even after getting through the errors, so I'm just going to ignore it for now.

I think I was able to get HDR working with GLES. (I'd need a way to take an HDR screenshot external to the application or a spotmeter to check the brightness coming out of the screen to verify.) So the answer might be EGL/Vulkan interop for Android if that's a thing. The Qualcomm developer page indicates Android only supports non-HDR Display P3 via Vulkan:
https://developer.qualcomm.com/sites/default/files/docs/adreno-gpu/snapdragon-game-toolkit/gdg/tutorials/android/hdr10.html#true-hdr-code-setup

Vulkan Swapchain/WSI for Android only supports VK_COLOR_SPACE_DISPLAY_P3_NONLINEAR_EXT.

For Windows, whether a sample should demonstrate HDR using WSI or DXGI has tradeoffs. HDR via WSI does not behave nicely, but it does "work" so it would make sense to show for Vulkan purity. On the other hand, most people will want DXGI in practice. I guess you could demonstrate both for completeness. A sample will probably also want to demonstrate handling window move/resize across SDR/HDR monitors, HDR on/off setting changes from the OS, and querying display capabilities.

[mirh]

Qualcomm also reports the same thing.. And I now found some actual official sample. I think setup should work just like with WCG, except you need android:colorMode="hdr" in the manifest and/or call setColorMode(ActivityInfo.COLOR_MODE_HDR) instead?

Did somebody even check my reply?
The android platform sample seems pretty clear, and they also added another doc in the meantime.
https://developer.android.com/media/grow/ultra-hdr-display#configure-window

HDR via WSI does not behave nicely, but it does "work" so it would make sense to show for Vulkan purity.

I guess that checks out.
mpv-player/mpv#12539 lists some further downsides then.

[alecazam]

According to this, on Android this is still wide gamut format. But Vulkan has zero docs on scRGB vs. what this color space actually means.

VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT - VK_FORMAT_R16G16B16A16_SFLOAT

https://developer.android.com/training/wide-color-gamut

[MaverickAl]

Some things that would be good to see in such a sample:
Taking screenshots
Calibration
Correction for SRGB_LINEAR_EXT

There's some good references for HDR10, e.g Retroarch, but they've had an issue about not being able to take screenshots open for some time.

Is this sample a priority or in the backlog?

[SaschaWillems]

I'm still working on it. My spare time is limited though.

Also please don't expect that sample to fit all your needs. There's a lot of stuff in here which is far beyond the scope of the samepl (e.g. calibration).

[YellowOnion]

Also many "HDR" monitors really aren't. They are 8-bit panels, or have no local dimming, have low nits (200-400) or require switching the Nvidia control panel from RGB4 to YCbCr422 to still run HDR10 at 60Hz (Samsung M8). So that's less color fidelity. But televisions seem to be getting this right, and newer monitors should fix this (Tempest G27).

This is incorrect, most modern computer LCD's are, yes only 400 nits and have 1:1000 contrast ratio, but these are not SDR displays, they're 4x brighter and 3x better contrast ratio than assumed in Rec 709 and sRGB, and plenty of displays have been using 6bits for SDR content and looked fine, coupled with PQ's far more efficient gamma curve, and software driven brightness, You will get a better image quality, with basic VESA verified HDR400 display, running in HDR mode, than running it in sRGB gamma and SDR mode with a static brightness or fake dynamic contrast mode with zero metadata.

The change from sRGB gamma to PQ will be the best singular thing you could do for quality, I remember seeing banding on Halo 1 with a 15 inch CRT.

And of course a 4k monitor that only has USB 3.1 at 5gbps would need chroma subsampling! I wouldn't conflate this singular bad monitor example with the plethora of other monitors that have HDMI 2.0 or DisplayPort 1.4 connectors that can handle the data rate requirements for 4k 10bits at 60fps.