[PoC] Steam overlay support for Gamescope WSI #1537
Comments
|
v2 - HDR works, overlay works char* hack = getenv("OVERLAY_HACK");
std::time_t start = std::time(0);
bool canBypassXWayland() {
auto secondsSinceBoot = (int) (std::time(0) - start);
if (hack != NULL && hack == "1"sv) {
auto decision = secondsSinceBoot > 10 || secondsSinceBoot < 5;
fprintf(stderr, "[Gamescope WSI] canBypassXWayland: since boot %d return %s.\n", secondsSinceBoot, decision ? "true" : "false");
return decision;
} |
|
The Steam Overlay does not support hooking Wayland. The correct solution is for the Steam Overlay to start doing its Gamescope mode for desktop probably. Adding delay hacks like this isn't a real solve. |
It will be when I finish the patch :) it can work synchronously without hardcoded delays
it does not matter since Proton is not creating Wayland windows, Steam Overlay is just looking for a created X11 window to properly enable itself, and will continue working even after the swapchain is swapped back to Gamescope WSI and the original window is gone If they created just a hack to hook CreateWaylandSurfaceKHR, it would be useless outside of Gamescope and entirely broken on desktop. Even if Valve wanted to support Wayland overlay injection, there is no API to do so while eliminating all X11 usage, it would still be needed to capture hotkeys and other things (majority of steam overlay functions as a vulkan post-render overlay, it's only the injection logic that's problematic) |
|
v3: now synchronous without hardcoded delays
diff --git a/layer/VkLayer_FROG_gamescope_wsi.cpp b/layer/VkLayer_FROG_gamescope_wsi.cpp
index c817880..a28945f 100644
--- a/layer/VkLayer_FROG_gamescope_wsi.cpp
+++ b/layer/VkLayer_FROG_gamescope_wsi.cpp
@@ -224,6 +224,10 @@ namespace GamescopeWSILayer {
}
}
+ const char *fixOverlayEnv = getenv("GAMESCOPE_WSI_FIX_OVERLAY");
+ if (fixOverlayEnv && *fixOverlayEnv && atoi(fixOverlayEnv) != 0)
+ flags |= GamescopeLayerClient::Flag::SteamOverlayFix;
+
return flags;
}
@@ -315,7 +319,7 @@ namespace GamescopeWSILayer {
xcb_connection_t* connection;
xcb_window_t window;
- GamescopeLayerClient::Flags flags;
+ GamescopeLayerClient::Flags flags = 0;
bool hdrOutput;
// Cached for comparison.
@@ -335,7 +339,19 @@ namespace GamescopeWSILayer {
return hdrOutput && hdrAllowed;
}
+
+ bool createdFirstSwapchain = false;;
+
bool canBypassXWayland() {
+ if (flags & GamescopeLayerClient::Flag::SteamOverlayFix && createdFirstSwapchain) {
+ // When creating first swapchain, we can't just return false, we need to avoid breaking HDR support detection in games as X11 rejects HDR formats
+ // canBypassXWayland() at boot will flip in sequence: true-false-true which will
+ // - allow games to detect HDR formats at start
+ // - allow steam overlay to inject by creating a working X11 window once (switch to SDR for a second)
+ // - destroy the X11 window and recreate a HDR swapchain, but steam overlay at this point is injected and will continue to work
+ return false;
+ }
+
if (isWayland())
return true;
@@ -1113,6 +1129,7 @@ namespace GamescopeWSILayer {
}};
pPresentModesCreateInfo->presentModeCount = uint32_t(s_MailboxMode.size());
pPresentModesCreateInfo->pPresentModes = s_MailboxMode.data();
+
return true;
});
@@ -1213,6 +1230,14 @@ namespace GamescopeWSILayer {
uint32_t(pCreateInfo->preTransform),
uint32_t(pCreateInfo->clipped));
+ if (!gamescopeSurface->createdFirstSwapchain) {
+ // First swapchain
+ gamescopeSurface->createdFirstSwapchain = true;
+ } else {
+ // Second swapchain or older
+ gamescopeSurface->flags &= ~GamescopeLayerClient::Flag::SteamOverlayFix;
+ }
+
return VK_SUCCESS;
}
diff --git a/src/layer_defines.h b/src/layer_defines.h
index 6ebfd32..c77b1f0 100644
--- a/src/layer_defines.h
+++ b/src/layer_defines.h
@@ -13,6 +13,8 @@ namespace GamescopeLayerClient
static constexpr uint32_t NoSuboptimal = 1u << 3;
static constexpr uint32_t ForceSwapchainExtent = 1u << 4;
+
+ static constexpr uint32_t SteamOverlayFix = 1u << 5;
}
using Flags = uint32_t;
-}
\ No newline at end of file
+} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As we know, steam overlay is broken unless Gamescope WSI is disabled, which is required for HDR gaming.
So currently steam overlay loads and can capture inputs just fine, but when you toggle the overlay it captures the cursor but the overlay stays invisible.
This can be fixed by delaying canBypassXWayland() to let it find the window in whatever unknown to me way steamoverlay does it
so at first the game is creating an SDR swapchain, then after 10 seconds it properly recreates a HDR one.
The overlay continues to work from that point.
Unfortunately, due to what I believe is not a gamescope problem but an Unreal Engine 4 limitation, it probably caches HDR support status early on(sees X11 rejects it), and the new HDR swapchain contains color-broken data from this point.
There's probably a smart way to do this without breaking HDR detection, like redirecting X11 window creation to creating dumb transparent window for steamoverlay to find, while still creating an HDR swapchain from the start.
The text was updated successfully, but these errors were encountered: