[rmodels] Optional GPU skinning

[orangeduck]

This adds a form of optional GPU skinning to raylib that doesn't interfere with the existing CPU skinning method, but does expose enough information for a user to write their own GPU skinning in shader code if they want.

Essentially the change works as follows:

• For all animated meshes the boneIds and boneWeights vertex data are now uploaded to the GPU along with the rest of the vertex data.
• Meshes contain a new boneMatrices array which is uploaded to the shader as a new uniform and is used to contain the skinning bone transformation matrices for that mesh. These matrices can be updated using the UpdateModelAnimationBoneMatrices function which essentially acts as a replacement to the UpdateModelAnimation function when you want to do GPU skinning.

And that is about it. The normal CPU skinning continues to work as intended where users call UpdateModelAnimation. However with this update users can instead call UpdateModelAnimationBoneMatrices and write their own shader which does the skinning. That might look as follows:

#version 300 es

in vec3 vertexPosition;
in vec2 vertexTexCoord;
in vec3 vertexNormal;
in vec4 vertexColor;
in vec4 vertexBoneIds;
in vec4 vertexBoneWeights;

#define MAX_BONE_NUM 128
uniform mat4 boneMatrices[MAX_BONE_NUM];

uniform mat4 mvp;

out vec3 fragPosition;
out vec2 fragTexCoord;
out vec4 fragColor;
out vec3 fragNormal;

void main()
{
    int boneIndex0 = int(vertexBoneIds.x);
    int boneIndex1 = int(vertexBoneIds.y);
    int boneIndex2 = int(vertexBoneIds.z);
    int boneIndex3 = int(vertexBoneIds.w);
    
    vec4 skinnedPosition =
        vertexBoneWeights.x * (boneMatrices[boneIndex0] * vec4(vertexPosition, 1.0f)) +
        vertexBoneWeights.y * (boneMatrices[boneIndex1] * vec4(vertexPosition, 1.0f)) + 
        vertexBoneWeights.z * (boneMatrices[boneIndex2] * vec4(vertexPosition, 1.0f)) + 
        vertexBoneWeights.w * (boneMatrices[boneIndex3] * vec4(vertexPosition, 1.0f));
    
    vec3 skinnedNormal = normalize(
        vertexBoneWeights.x * (mat3(boneMatrices[boneIndex0]) * vertexNormal) +
        vertexBoneWeights.y * (mat3(boneMatrices[boneIndex1]) * vertexNormal) +
        vertexBoneWeights.z * (mat3(boneMatrices[boneIndex2]) * vertexNormal) +
        vertexBoneWeights.w * (mat3(boneMatrices[boneIndex3]) * vertexNormal));

    fragPosition = skinnedPosition.xyz / skinnedPosition.w;
    fragTexCoord = vertexTexCoord;
    fragColor = vertexColor;
    fragNormal = skinnedNormal;

    gl_Position = mvp * vec4(fragPosition, 1.0f);
}

I have tested this in my own little demo but I have not yet run through to see if it breaks anything else or to check it works with all the existing demos. That is still to do!

I know your intention is to keep raylib simple and minimal and this update is a little hacky in some senses so obviously feel free to close it if you think it is not the right approach or not useful. I just thought I would share it in case you think it might be valuable.

Many thanks in advance!

Dan

[orangeduck]

I checked and currently this does break the examples and the existing model rendering when no shader is supplied. I'm investigating why now.

[orangeduck]

The issue appeared to be inserting the new shader uniform locations into the middle of the shader locations enum. Perhaps someone with more experience can move them into a more logical position but I'm not sure right now what may be hard-coded in raylib which relies on the current locations and needs updating. I will also try to prepare a small example for the examples folder.

[orangeduck]

Okay I added a small example models_gpu_skinning and merged with the latest changes. Let me know if there are any other changes I can make - or as I mentioned before feel free to close this if it is not an approach you think raylib should take in the end :) Thanks!

[raysan5]

@orangeduck Hi! Thank you very much for this improvement! It has actually been requested many times by raylib community! The main concern it was not added before was the lack of support for UBOs by some OpenGL versions but as per my understanding, this solution can work with any OpenGL version.

Just let me some days to review it more carefully, specifically the ShaderLocationIndex order issues and the function naming. Also a bit concerned about the Model structure size increment...

In any case, I'm really confident on merging this improvement, after all you are probably the developer that has pushed further raylib 3d models animations!

[orangeduck]

@raysan5 Brilliant, thank you!

Yes I think this can work on any OpenGL version. I believe the main limitation to this approach is the uniform mat4 boneMatrices[MAX_BONE_NUM]; uniform. For some platforms which have limited memory for uniforms MAX_BONE_NUM may need to be relatively small (my laptop GPU seemed okay with 128, but could not handle 256). For this reason I have seen some developers use a texture or a uniform buffer object instead - but then we are back to OpenGL version issues - so perhaps given raylib is already limited to 256 bones by the boneIds array this may be a practical trade-off for most people.

The other downside is of course that for animated models that use CPU skinning you will now pay the additional VRAM cost for uploading the boneIds and boneWeights.

Thanks again!

[raysan5]

The other downside is of course that for animated models that use CPU skinning you will now pay the additional VRAM cost for uploading the boneIds and boneWeights.

Well, that can be addressed with a config.h flag, for example SUPPORT_MESH_GPU_SKINNING. Only upload boneIds and boneWeights to VRAM if enabled and use CPU if not.

Also, in case it is enabled + we are on a limited platform (OpenGL ES 2.0) + an animated model loaded has >MAX_GPU_BONE_COUNT, a warning TraceLog() message can be shown, just in case...

[raysan5]

@orangeduck Hi Daniel, it seems this could be a potential issue, this variable should be probably k.

[orangeduck]

Thanks! I took a look and I think it does look safe to use j here I just need to remove the variable declaration as all the iterator variables are declared at the top of the function.

[raysan5]

@orangeduck About to merge this great improvement, just a small issue detected by the automatic analysis system.

[raysan5]

@orangeduck Thank you very much for this great improvement! 😄