Add support for sharing and preprocessing shader code (through #include, #if, #ifdef, …)

[Zylann]

Continuation from godotengine/godot#11691

Describe the project you are working on:
A terrain plugin with lots of options

Describe the problem or limitation you are having in your project:
Everytime I add a new kind of shader, or a variation of one (with less uniforms for example, or a different usage), I have to copy/paste the entire shader and alter the 10% specific parts. This is a burden to mainatin all the copies.
Besides, even in other projects with shared dependencies, it is also mandatory to copy entire functions.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
I need a way for a shader to specify other files containing code on which it depends on. It would allow to share at least functions and constants, so that would reduce code duplication a lot. It would make it much easier to create libraries as well.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
Just like you can have a resource depend on another (i.e in GDScript you have preload), a new instruction can be added to shaders in order to include code in the final source to compile.
This is typically achieved with a preprocessor directive such as #include "path/to/shader.shader".
Of course, cyclic inclusion is forbidden, but nesting would work, and would be tracked recursively so if a shader included into another is edited, it would update shaders it is included from.

One problem is the fact shaders included in this way cannot really have the same structure. For example, you would not be able to use render_mode, shader_type, fragment or vertex, since they would collide with the "root" shader. You'd be able to define functions, constants, and perhaps even uniform and varying, since those are user-defined symbols.
To differenciate those "includable" shaders, they can be given a different extension, such as shaderinc, or shaderlib, which reflects their purpose.

Why not global shaders:
I recall seeing a proposal of global shaders. They might be "global" for another reason, but if one goal is to have them share their functions with every other shader in the project, it has huge potential for name-space pollution, conflicts and increasing compile times, so I would not recommend this. With targeted dependencies, you can decide where you get the shared code, keep it self contained and non-intrusive.

Visual Shaders:
the same applies, as long as those shaders can declare functions, constants, varyings and uniforms. But since Visual Shaders are code generators under the hood, they might be able to do code sharing in their own way.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
No. You might be able to setup a tool pipeline with a plugin to make shader inclusion working, but it's a significant amount of work and I'm not sure if it will integrate well. It requires shader resources to have defined dependencies and plugins cannot add that. All they can do is to define different types of non-shaders for which they have to recode a whole editor for, and would bake final shaders which would then be read-only and not modifiable in realtime.

Is there a reason why this should be core and not an add-on in the asset library?:
Quality of life, basic feature, hard to integrate as a plugin. It's a recurring problem when your project starts having more than a few shaders, especially when you design a library for other people to use. Besides, as seen in the linked issue, it has been requested and PRed several times.

[Calinou]

As a workaround, you could look into integrating a templating system like gomplate to your project. This way, you can generate all variations with a single command.

[clayjohn]

I wonder if this could piggyback on the per-instance uniform/global uniform work.

[SIsilicon]

This can be part of a pull request along with the other preprocessor directives (#define, #if, etc..)

[Xrayez]

This can be part of a pull request along with the other preprocessor directives (#define, #if, etc..)

I feel like this could be especially relevant since the drop for GLES3 in 4.0 or similar use cases (also GLES2 / GLES3 differences), so let me link it here:

• Preprocessor support for Godot shader language godot#17303.

[SIsilicon]

There is something bothering me on how the preprocessors should be implemented... #include is relatively easy. Just load the shader resource into the line of the other shader (while counting for line changes when throwing errors). The other preprocessors however are different.

[Lywx]

Actually this is quite easy task. There are a number of very robust preprocessor libraries like Boost Wave could be helpful for this feature.

[Lywx]

You could easily support all standard C++ pre-processor like #define #ifndef etc.

[Calinou]

Actually this is quite easy task. There are a number of very robust preprocessor libraries like Boost Wave could be helpful for this feature.

Don't use complex canned solutions for simple problems 🙂

[ChristianB84]

At least on my machine, using uniforms as upper bounds of for-loops often is by far less efficient than using constants, even if I set the uniform to the same or even a lower number. I guess this is because the loops can't be unrolled during compilation.

I mostly want to change the upper bounds of for-loops for performance settings, such that, e.g., more or less steps for parallax mapping can be configured. In that case, it'd make sense to re-compile the shader with a new constant whenever the setting is changed, because that doesn't happen very often.

[adamscott]

There is something bothering me on how the preprocessors should be implemented... #include is relatively easy. Just load the shader resource into the line of the other shader (while counting for line changes when throwing errors). The other preprocessors however are different.

@Calinou, shouldn't the proposal be split in two? As much shader recompilation/macro conditions are nice features, the ability to include/compose shader code answers a different problem, even if the implementation of both can be, at glance, very similar.

[Calinou]

@adamscott Probably, I'll let @Zylann do that if he's around.

[alexfreyre]

Another example very useful for this feature:

https://github.com/patriciogonzalezvivo/lygia

[jitspoe]

I've been using iFire's shader includes for quite some time in my custom 3.x branch, and it's been working great! I'm guessing the holdup is getting support in 4.x? Would be awesome to have this officially part of the engine.

[Calinou]

I'm guessing the holdup is getting support in 4.x?

Indeed, features must be implemented in master before they can be backported to 3.x (unless said feature isn't relevant for master). This ensures feature parity between 3.x and master.

[AlfishSoftware]

I never noticed this. @Calinou Could you please ping me when any .gdshader syntax update is implemented (unless there's somewhere I can subscribe to get notified specifically on those)? Is there any other change I might have missed?

[Calinou]

I never noticed this. @Calinou Could you please ping me when any .gdshader syntax update is implemented (unless there's somewhere I can subscribe to get notified specifically on those)? Is there any other change I might have missed?

Sorry, I can't keep track of changes and remember to individually ping anyone who asks me about it. I suggest looking for a service that lets you receive email notifications when a file in a GitHub repository is changed (in this case, https://github.com/godotengine/godot/blob/master/servers/rendering/shader_compiler.cpp).

[AlfishSoftware]

Well, thanks for pointing me to the file and the right direction. I'll see if I can set it up somehow.
I wish GitHub had this file-watching feature built-in.
EDIT: The commits page for a file or folder has an RSS feed, so I can subscribe to it by appending .atom in the URL.