GPU: Support swapchain buffer transparency in Vulkan

[Kontrabant]

If the window is flagged with SDL_WINDOW_TRANSPARENT, create the associated swapchain with a composite alpha value that supports blending, if available.

Fixes transparent windows on the Vulkan backend, and prevents possible validation errors by ensuring that the composite alpha value is always a valid bit in VkSurfaceCapabilitiesKHR::supportedCompositeAlpha.

To note, the DX 11 and 12 backends don't currently support transparent windows either. I tried implementing this for DX12, but all I was able to get was a black or empty window when using IDXGIFactory4_CreateSwapChainForComposition, even with all of the DXGI_SWAP_CHAIN_DESC1 struct parameters seemingly set correctly and the function succeeding. Looking at the old DX12 renderer, it looks like transparent windows just don't work with DX12?

SDL/src/render/direct3d12/SDL_render_d3d12.c

Lines 3202 to 3204 in f827c13

	if (SDL_GetWindowFlags(window) & SDL_WINDOW_TRANSPARENT) {
	// D3D12 removed the swap effect needed to support transparent windows, use D3D11 instead
	return SDL_SetError("The direct3d12 renderer doesn't work with transparent windows");

DX11 currently uses DXGI 1.0, but would need 1.2 to enable window transparency, which would require a bit of refactoring.

[thatcosmonaut]

I don't want to merge features that only work on one backend, that's how we make a hacky mess for clients.

We actually optionally use features up to DXGI 1.6 on DX11, you can see this in the QueryInterface calls. If you can get this feature working on all backends I will reconsider merging this. DXGI 1.4 is required to be supported on Windows 10+ so that's an incredibly wide range of systems. You would also have to look into whether Metal supports this.

[thatcosmonaut]

Also: if this feature is only conditionally available I would be in favor of adding a SupportsSwapchainTransparency function that the client can check before using this feature.

[flibitijibibo]

At least for Vulkan this should be okay - Metal has a similar flag from what I recall, but if we need to do these one at a time this is fine

[thatcosmonaut]

I just want to make sure this is at least possible on every backend first.

[e4m2]

I think the DirectComposition stuff should work on D3D11/D3D12 using WS_EX_NOREDIRECTIONBITMAP according to this MSDN article.
For this simple use case, you don't need to pass a DXGI device when creating the DComp device, which is important because AFAIK there is no way to query a DXGI device from a D3D12 device.
The unsuffixed creation function is not officially documented to accept NULL for the DXGI device, so it's probably better to use DCompositionCreateDevice2() instead.

You could quickly test if it works by replacing the current window transparency code

SDL/src/video/windows/SDL_windowswindow.c

Lines 741 to 762 in d1739ce

	// FIXME: does not work on all hardware configurations with different renders (i.e. hybrid GPUs)
	if (window->flags & SDL_WINDOW_TRANSPARENT) {
	void *handle = SDL_LoadObject("dwmapi.dll");
	if (handle) {
	DwmEnableBlurBehindWindow_t DwmEnableBlurBehindWindowFunc = (DwmEnableBlurBehindWindow_t)SDL_LoadFunction(handle, "DwmEnableBlurBehindWindow");
	if (DwmEnableBlurBehindWindowFunc) {
	/* The region indicates which part of the window will be blurred and rest will be transparent. This
	is because the alpha value of the window will be used for non-blurred areas
	We can use (-1, -1, 0, 0) boundary to make sure no pixels are being blurred
	*/
	HRGN rgn = CreateRectRgn(-1, -1, 0, 0);
	DWM_BLURBEHIND bb;
	bb.flags = (DWM_BB_ENABLE | DWM_BB_BLURREGION);
	bb.enable = TRUE;
	bb.blur_region = rgn;
	bb.transition_on_maxed = FALSE;
	DwmEnableBlurBehindWindowFunc(hwnd, &bb);
	DeleteObject(rgn);
	}
	SDL_UnloadObject(handle);
	}
	}

with setting the flag in GetWindowStyleEx() instead.

However, doing it this way almost certainly regress OpenGL and generally anything that doesn't use DXGI flip presentation. The linked article even suggests that using the composition API is required:

In this case, the window has no redirection surface, so the DXGI factory’s CreateSwapChainForHwnd method can’t be used.

But I've actually found this to be false, HWND flip swapchains generally do work without redirection surfaces (but the fancy transparency stuff is still composition-only).