Add user-defined grouping to Render and Physics layers

[YuriSizov]

Describe the project you are working on

A puzzle game that relies on areas to detect various forms of interactions.

Describe the problem or limitation you are having in your project

While squares/cells provide a quick way to set up collision layers and masks, they have a huge usability downside for me: I always forget which cell is which in that grid of 20 faceless gray/blue buttons. Yes, you can name layers, but that information is not readily available and requires you to either hover over each (which takes time and is generally cumbersome) or to use a dropdown menu. That menu is nice, but then what's the point of having these ease-of-use cells?

There is also a problem that we only allow to set 20 layer bits out of 32 available. Having even more cells there would make the grid of squares even harder to navigate (see #2770). So to help myself a bit I adopted a practice of using visual separation provided by the inspector to create "groups" of cells. I can use 1st and 2nd cell for one thing, and 11th for another type of collision. This helps separate them mentally and leaves space after 2nd to add more of collision types of the same quality at 3rd, 4th, etc positions.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

What can help here is, I believe, groups. If users are provided with a way to define groups for layers, which the cells can then be assigned to, we could display those groups in the inspector as a set of neat, collapsible sections. Groups can have their own name and color with latter providing better at a glance separation for different kinds of collision layers. Otherwise the control can work as now, just split between groups.

All layers that haven't been assigned a group are displayed in an "unsorted" group that is always available at the end of the inspector section. This group can be big (and will be the only group by default, so would have the same 20 to 32 cells as we have now). So for big groups we still do similar visual separation as we do presently. We can adjust from base-5 to base-4 as we increase the number from 20 to 32.

One could argue for coloring individual layers instead, but I think colored groups make more sense visually but restrict users from overindulging and creating a colorful mess. Colors usually have a special meaning in each system of organization, so it makes sense to associate color with a name, and thus have a colored group. As a bonus colors an later be somehow utilized when using collision shape debug options.

Groups would also allow users to break from bit/index order in which cells are currently presented, as layers from all over the spectrum can be used in the same group. This means no longer one needs to leave space in the cell grid for future-proofing.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The UI can be done in the way depicted in this rough mockup:

The configuration is done in the Project Settings dialog. Each layer gets a secondary setting after the name to set a group. Groups are defined in the same UI separately, they must be assigned a name and a color. This information should act as auxiliary data, so it shouldn't increase complexity of the inner workings of the layers. Instead if should be only used for visualization.

Changes in code would involve handling the creation of group items in the Project Settings dialog (not sure if we have anything like that currently), and enhancing the EditorProperty editor for layers. The latter part should be pretty trivial.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Not a few, but I guess I can imagine a plugin that can be used to override the inspector for these properties.

Is there a reason why this should be core and not an add-on in the asset library?

As mentioned, this can likely be an editor plugin, but I think that the current UX of setting collision layers is simply bad so a fix needs to be done in the editor and not as a third-party patch.

[Zireael07]

As I mentioned on the other issue, names/groups are necessary for usability (otherwise the names that we can set in settings are simply hard to access, basically making them serve no use). Colors are a nice-to-have but names are necessary.

As is, I tend to use layers sparsely and rely on groups even for physics stuff (as in, ignore collision if the collider is in group X or has name Y) because it's difficult to remember what each layer is for past 3-4 layers.

[pouleyKetchoupp]

It seems like a good idea! As an alternative that might be a bit easier to setup and have a more generic layout, we could have group names for blocks of 8 values, something like that:

Probably with group colors like you did and numbers in the squares as well to help make it more readable.

[groud]

As I said on RocketChat, I don't think adding group is needed for organizing the physics/render layers.

Right now, we can define a name per layer, which makes quite a lot of sense, Typically, in your example, I don't understand why you would need more than one collision group for "Items and power ups" or for "enemies". I think the most common use case would be to have one layer per type of game object, thus grouping them in a single layer. Though I am maybe wrong, but I'd like to see real life use case there.

However, I understand the issue about the fact organizing layers is difficult. I see several problems:

• It is difficult to identify each layer visually, in the bitmask form,
• Layer names are hardly accessible,
• The inspector gives you a lot more layers than you would need in real life use case.

Even if grouping might help a little bit here, I think it makes the layers management needlessly more complex.

I would instead suggest other options:

• Allowing to limit in the project settings the number of layers,
• Allowing to define custom colors for each layer. Since I don't think layer groups are very necessary (as there will likely be a lot of situations where groups will not contain more than one or two layers), I don't think it's an issue if people would like to group layer by reusing the same colors,
• Find a way to display layer names on hover, but with a popup very fast to appear, or even a label in the inspector.

That being said, @pouleyKetchoupp has an acceptable compromise. I still don't get the use case where you would need 8 layers in a group, but at least it does not make the layers system a lot more complex.

[YuriSizov]

Though I am maybe wrong, but I'd like to see real life use case there.

In my projects the use case is pretty obvious: I have a single type of object, but different types of areas it might have for detecting different interactions. So I would have a group named "Part", for objects of the type Part, and it would have layers for detecting one type of interactions, then another type of interaction, then yet another, etc. Then I'd have a group named "Blob", similarly for the Blob objects, and they may have different areas to detect different things.

You may be thinking in terms of pure physics, but I don't use these for physics. I use areas for other types of collisions, with other areas or rays, and even in a tiny game I've needed to have several types of areas on the same type of object. Sure, it should be possible to code some logic to work around this while using only one layer per object type. But why? We have 32 bits available, and that type of ease of separation are what they are there for. So I don't see it hard to imagine wanting to group your layers.

[groud]

You may be thinking in terms of physics, but I don't use these for physics. I use areas for other types of collisions, with other areas or rays, and even in a tiny game I've needed to have several types of areas on the same type of object. So I don't see it hard to imagine wanting to group your layers.

Yeah, but I mean, this is kind of a misuse then. Compared to collisions where you will need different layers to have different behaviors, you can a lot more easily use a bunch of if/else in your detection area's script to choose whether or not you should have an interaction with the detected object. Unless you have hundred of overlapping areas, there's not a lot of situations where the performance gain given by the physics system will outweigh the flexibility provided by scripts.

[YuriSizov]

How's writing a script easier than preventing an event from happening to begin with, especially when we have a tool for that? Why is that a misuse? Are you saying that layers should never be used for logic? Why?

[groud]

How's writing a script easier than preventing an event from happening to begin with, especially when we have a tool for that? Why is that a misuse? Are you saying that layers should never be used for logic? Why?

Oh, well, you can use it, but the fact it is limited by an arbitrary number or that it is more difficult to change over time (there's no way to reorder easily the layers for example) makes me think that using physics layers for that kind of logic is less than ideal.
And, if I am not mistaking, adding some logic with script seems to be to most common way out of the two (at least from the tutorials I've seen).

[dalexeev]

So I don't see it hard to imagine wanting to group your layers.

If the checkboxes remain unsigned inside the layer groups, then this is not much different from the current system. You still must remember what the first layer in the Enemies group is responsible for, for what the second, and for what the third. If you sign each checkbox, then the layer groups become less important. Signed checkboxes are already in the popup.

[YuriSizov]

If the checkboxes remain unsigned inside the layer groups, then this is not much different from the current system.

It is a huge difference to me. You can at least have visible collections of squares that are in fact in use. You can tell from the name what they are used as. Can you tell me any of this from this UI?

Or even from this?

Well, maybe if I have just 3-4 layers in the whole project, but not if I use more of them, all of them. Why have a 32 bit mask at all, if the proper way to use layers is not to use them and write everything in scripts? I don't get it.

[dalexeev]

I do not argue, the text gives more information. But what can I say looking at this?

Then you need to go further and be able to see the name of each layer in the inspector, and not just the group of layers. And instead of constantly collapsing/expanding layer groups in the inspector, it is much faster to open a popup with signed checkboxes or see a tooltip.

It seems to me that the property in the inspector just has a slightly different purpose. You don't mind the fact that you need two clicks to edit the align property instead of one?

[YuriSizov]

But what can I say looking at this?

That was just to show how a bigger group should look like, I try to demonstrate permutations with my mockups. Whether someone would achieve this state or not is a separate question.

Then you need to go further and be able to see the name of each layer in the inspector, and not just the group of layers. And instead of constantly collapsing/expanding layer groups in the inspector, it is much faster to open a popup with signed checkboxes or see a tooltip.

It seems to me that the property in the inspector just has a slightly different purpose. You don't mind the fact that you need two clicks to edit the align property instead of one?

My proposal is unrelated to the use of squares. I would still like a way to organize layers even if we end up with a dropdown list of checkboxes. My proposal is about organization, not about a specific way to make the selection. I only use current design to demonstrate how it can look. If #2770 ends up being resolved in a manner that changes the look and behavior, then I can adjust my mockups to reflect that.

[2plus2makes5]

Your mockup made me think that a maybe simpler solution could be to show only layers being used, for example if we are using 3 layers than the editor shows only 3 squares, and setting different colors for them. The number of layers and their colors could be chosen in the project settings.

[YuriSizov]

@2plus2makes5
I don't think we can check which layers are being actually used (that would require parsing all the scenes, plus these values can be set in code), if that's what you mean. But if we were to make this form more concise in general, I'd go for "show only named layers". This is more up the #2770 valley though.

[2plus2makes5]

@pycbouh sorry i guess i wasn't clear, i'm not suggesting to automatically check the number of layers used, but to set it in the project setting.
In #2770 i really like your last mockup with numers in squares and names shown and the one of dalexev.

[YuriSizov]

In #2770 i really like your last mockup with numers in squares and names shown

That's not mine, but I also like it 🙃

but to set it in the project setting

In that case I think it should be enough to rely on the name being set. I think pretty much everyone must name them if they use them, so having a name is similar to having them checked.

[dalexeev]

I agree, it would be nice to distinguish between the layers. Checkbox numbers, instant hints, and colored checkboxes can help solve this problem. But IMO, layer groups are redundant. Because by themselves they do not allow you to distinguish between layers, and when this problem is solved, the need for groups decreases dramatically. As a last resort, you can, for example, use prefixes:

Group A
    Layer 0
    Layer 1
Group B
    Layer 2
    Layer 3

Group A - Layer 0
Group A - Layer 1
Group B - Layer 2
Group B - Layer 3

Since layer groups do not affect the behavior of layers, but are only for the convenience of the developer, layer groups are not needed. There are a limited number of layers, there are only 32, not 100.

Also, layer groups are not a very common use case. The layers have been designed to be unique, two layers should not repeat each other.

In many cases, you can do without layers altogether, and use node groups, the is keyword, or duck typing (checking for a method). Layers should be used mainly only when different bodies should not collide, but pass through each other. Heavy use of Layers for Area2D(3D) is a vicious practice.

[YuriSizov]

Groud also suggested prefixes on the RC yesterday, but that's not a good way to solve UX problems by making users create workarounds. Plus, prefixes don't help group layers. You still need to leave space between indices for future-proofing. So it won't be

Group A - Layer 0
Group A - Layer 1
Group B - Layer 2
Group B - Layer 3

It will be

Group A - Layer 0
Group A - Layer 1
...
Group B - Layer 7
Group B - Layer 8

... in case you want to extend the first group.

Since layer groups do not affect the behavior of layers, but are only for the convenience of the developer, layer groups are not needed.

I fail to see any logic in this statement. "Since feature is needed for convenience, feature is not needed." What exactly are you arguing here? That we shouldn't improve user experience because it's only for convenience? Remember as well that Godot editor is supposed to be used by various types of game developers, not just coders. We have a whole system to disable features, even individual classes from appearing in the UI just so that the editor can be tweaked to provide the best UX to a specific type of a person on your team. We have in-editor tooltips for nodes in scenes with user descriptions, just for small convenience, especially when collaborating with other people. This is hardly the first thing we may add just for the sake of making the editing process more clear.

Also, layer groups are not a very common use case. The layers have been designed to be unique, two layers should not repeat each other.

I don't know where you've got an idea of something repeating, I've never given anything like that as an example.

Layers should be used mainly only when different bodies should not collide, but pass through each other.

This is exactly what I use Areas on different layers for. It just that neither you nor groud can appreciate for some reason that the same type of object can have different layers to detect different things and ignore other things that object may interact with. Sure, there are many other ways to patch the holes in this boat, but I don't understand why both of you claim that using layers within the allowed limits is a bad or a vicious practice. Seems like a pretty arbitrary limit invented just for the sake of arguing.

[dalexeev]

Groups imply that there are so many (possibly unlimited) elements that they cannot be memorized. How many groups can consist of 32 layers, if one group should contain more than one layer, more likely even more than two, but not too many, and each layer can be included in only one group? Answer: 8 groups of 4 layers and they stand out visually well. It is unlikely that you will have more than 8 groups, and it is unlikely that most of them will have more than 4 layers.

I fail to see any logic in this statement. "Since feature is needed for convenience, feature is not needed." What exactly are you arguing here?

The fact of the matter is that the convenience is not so obvious. The concept with expandable tabs didn't appeal to me at all. In the current version, you can just click once, in the proposed one you need to constantly collapse/expand. At the same time, I do not deny the problem with distinguishing between layers (and I like the suggestions from #2770), but I think that groups are an overkill for such small numbers as 20 and 32.

this is exactly what I use Areas on different layers for.

the same type of object can have different layers to detect different things and ignore other things that object may interact with

Opinions vary here, but it seems to me that areas shouldn't ignore anything (as long as it doesn't affect performance). Areas can filter bodies in scripts (perhaps the best way is duck typing). Collision layers, as their name suggests, are for bodies so they can ignore and pass through each other. When collision layers start to be used in areas to control logic (so as not to check the object entering the area), this is bad.

[YuriSizov]

The fact of the matter is that the convenience is not so obvious. The concept with expandable tabs didn't appeal to me at all. In the current version, you can just click once, in the proposed one you need to constantly collapse/expand. At the same time, I do not deny the problem with distinguishing between layers (and I like the suggestions from #2770), but I think that groups are an overkill for such small numbers as 20 and 32.

I can see what you mean here, thank you for clarification. Honestly, I don't think this needed to be a separate proposal, I was just trying to give you a different outlook on #2770 since it seemed like a whole day passed bike-shedding how many squares in a row is too much. Groups, or just colors, can be implemented in many different ways in the end, so I don't want anyone to focus on the way I made this particular mockup.

How many groups can consist of 32 layers

I can only extrapolate from my experience on smaller projects, so it seems that you can need more than 2 or 3 groups there. But I have no real data on that scale to back me up.

perhaps the best way is duck typing

I don't like relying on duck typing if I can help it. This can be a solution to some, but there are many way to organize your codebase, and while we do not have interfaces in GDScript I'd prefer to solve this otherwise.

When collision layers start to be used in areas to control logic (so as not to check the object entering the area), this is bad.

But why? This is exactly what in-world triggers are in all games: an area (or a box in 3D) with a collision shape that signals that something needs to happen when something else enters it. Take Source, for example, that has boxes that define various parts of the level logic. Of course I don't know where the separation is made in their code base, but from the level designer perspective you create boxes that act for specific logical interactions, some with physics involved (like grenade bounces), some without (like death triggers).

If Area2Ds and CollisionShapes are not appropriate for this function, what in Godot is? Manually working with global coordinates and Rect2s?

[dalexeev]

I can't say exactly where the border goes, but you need to avoid using layers for everything, trying to protect Area2D from being triggered. Compare:

walls, characters, bullets, power_ups

walls, ground, player, allies, foes, player_bullets,
allies_bullets, foes_bullets, coins, boxes, savepoints

I don't like relying on duck typing if I can help it. This can be a solution to some, but there are many way to organize your codebase, and while we do not have interfaces in GDScript I'd prefer to solve this otherwise.

Duck typing are dynamic interfaces (also known as interfaces without guarantees or interfaces without interfaces). But either use that or use node groups, which I don't really like either.