Ensure RigidBodies only interact with Bodies with layers in their mask

[madmiraal]

The RigidBody sister PR to #42268 (Areas) and #42641 (KinematicBodys).

Currently, when a RigidBody collides with another CollisionObject the collision_layer->collision_mask combination is checked both ways. This results in unintended and unwanted collisions.

This PR ensures that RigidBodys only collide with CollisionObjects that have one of the RigidBody's collision_layer bits in the other CollisionObject's collision_mask. It fixes both 2D and 3D Godot physics.

Note: Unlike #42268 and #42641, this PR does not apply the same fix to Bullet physics.

The final part of godotengine/godot-proposals#2775.

[pouleyKetchoupp]

I haven't tested yet, but I would expect that when both bodies are dynamic and one of them ignores the contact, its mass is considered infinite (like a static/kinematic body) in order to correctly apply impulses on the other one. Otherwise the bodies are going to go through each other.

[pouleyKetchoupp]

It would be nice to have a member or at least local variable to avoid repeating the same logic multiple times.

[madmiraal]

I prefer it this way. It's the same format as the other condition, it's method name makes it easy to understand when reading, and the compiler will inline it anyway.

[madmiraal]

I would expect that when both bodies are dynamic and one of them ignores the contact, its mass is considered infinite (like a static/kinematic body) in order to correctly apply impulses on the other one. Otherwise the bodies are going to go through each other.

They should go through each other. If a CollisionObject is not in a layer that is in this CollisionObject's mask, then the first CollisionObject should be ignored.

When I originally created #42641, I used the following test project, created in 3.2 to test a combination of #42642 and #49169 2D and 3D: 15243.zip.

The following videos show the output of those tests
Current 2D i.e. without #42642 and #49169:

2D with #42642 and #49169:

Current 3D i.e. without #42642 and #49169:
Note: The 3D box layout and colours are the same as the 2D so use the labels in the 2D versions above to confirm the expected outcomes.

3D with #42642 and #49169:

[pouleyKetchoupp]

@madmiraal You can see in the 2d version how rigid body 1 slightly goes into rigid body 2, my guess is it's a buggy collision that is due to what I'm talking about and would be fixed with my suggestion.
We can't design a system where some object will half collide, from the user point of view an object should either collide or go through.

[pouleyKetchoupp]

After more investigation on the topic of contact impulses, here's my conclusion:

The visible overlap in 2D I mentioned before is not actually due to the problem I'm pointing out, it's because the sprites are larger than the collision shapes in your test scene, that's why it also occurs in the first gif.

The issue I'm talking about is not visible in your scene because collisions, motions, shapes are very simple, but it has the potential to cause problems in more complex scenarios.

Applied impulses in one solver iteration are not physically correct
I've made some tests in 3D based on your test project to illustrate this issue.

In this scenario, RigidBody1 is falling on top of RigidBody2, and layers are set in this way:
-RigidBody1 is affected by RigidBody2
-RigidBody2 is not affected by RigidBody1

Here's the overlap before the solver kicks in to handle contact impulses:

If the rigid bodies collided normally, here's what you would get in one solver iteration:
(both would be pushed from each other)

And here's the comparative result of this PR after one solver iteration:

This PR	Proposed fix

You can see with the current state of this PR, RigidBody1 is not pushed out enough in one solver iteration, because RigidBody2 is expected to move too during impulse calculations, and in the end only RigidBody1 actually receives an impulse.

The fix is not complicated, it just consists in considering RigidBody2 kinematic (infinite mass and infinite inertia) when it comes to interacting with RigidBody1. There's no negative side effect and it can only improve physics for a similar case in more complex scenarios.

[madmiraal]

RigidBodies not fully recovering their penetration with each other is an old, pre-existing issue (#2092, #16505, etc.) that this PR is not aiming to address.

[madmiraal]

Updated to include equivalent fix from #50685.

[pouleyKetchoupp]

Superseded by #50625.