Discussion: Shader compile logging/analysis/tracking

[kg]

This is kind of an SDL_gpu thing but also relevant to the design of the shader compiler and infra.

Right now in my game and libraries I have a bunch of infrastructure designed to ensure that shader compiles happen at loading screens instead of during actual rendering. This is pretty tricky to get right - you need to make sure that you get things like vertex and fragment shader inputs, samplers etc are all configured correctly to match how the shader will actually be used, then do a draw with it so that the driver warms everything up.

The ideal would be to have something flow all the way through such that the user can register a logging callback that tells them every time a shader compile happens, either a driver compile or a SDL_gpu compile or whatnot. That way during debugging and QA you can ensure that there aren't runtime compiles happening during draw, and you know which shader is responsible when it happens.

I think this would not demand too much in the shader compiler - just make sure annotation data is tracked, so that at the right time we can figure out the identity of the SDL_gpu shader and include it in the logging.

The SDL_gpu API itself would also need to flow information through so that when the relevant compile APIs get called, we know who to blame.

There are some existing tools out there you can use to audit this stuff as a developer - IIRC some OpenGL drivers will tell you about this, and vulkan validation layers can also help. For D3D the best way to find it is in a profiler since the compile operations are really slow :-)

(I guess one other related topic would be making sure that when the SDL shader compiler generates other formats, like GLSL/HLSL/DXIL/SPIR-V, names and such flow through so that you can identify shaders in tools like RenderDoc. Right now when using FNA3D+MojoShader, every shader is named something like 'ShaderFunction12', so this is an area that the new SDL shader compiler could really improve on 😊)

[icculus]

I never replied to this before; sorry about that.

The hope is to pass as much information through to the lower-level as possible (which is why most things at the C API level take an optional name string, and eventually we'll have optional debug tables in the bytecode). The hope is to expose these things to debug and profiling tools when we can.

I'm not against finding a way to hook in and alert when the lower level has decided to actually compile a D3D/SPIR-V/whatever shader, but I haven't thought about this in great detail yet. But it seems like a reasonable thing to do. Bytecode to lower-level shader happens at a specific, well-defined point in SDL, but that's not much help when drivers do all sorts of magic to delay the low-level compile.

On my big fantasy wishlist is hooking up RenderDoc directly to this API, so you aren't looking at Vulkan calls, you're looking at SDL GPU calls. But that might be a nightmare to actually accomplish...but I really want it.

[kg]

Happy to hear the plan to eventually have debug tables in the bytecode. Having that kind of information has been really useful for dev tools and diagnostics in the past, like automatically verifying that CPU side constant buffers match shader layouts, etc.