Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Meshlet GLTF processor #13431

Closed
wants to merge 39 commits into from
Closed

Meshlet GLTF processor #13431

wants to merge 39 commits into from

Conversation

JMS55
Copy link
Contributor

@JMS55 JMS55 commented May 20, 2024

Objective

  • Allow importing GLTF scenes (gltf + separate bin, gltf embedded, or glb) and converting all meshes to MeshletMeshes

Solution

  • bevy_gltf has two new features - meshlet, and meshlet_processor. The former is needed for loading processed GLTF files containing MeshletMeshes. The latter is needed for the conversion process (with the asset_processor feature).
  • RawGltfLoader - Similiar to GltfLoader, but returns the underlying gltf-rs Gltf json structure in a RawGltf, instead of parsing it into bevy's Gltf type (which stores handles to meshes/materials/images/etc).
  • MeshletMeshGltfSaver - Takes a RawGltf in, converts each mesh primitive to a MeshletMesh, adds the BEVY_meshlet_mesh extension to the primitive which points to a slice within buffer 0 (the GLB BIN buffer), and then appends the serialized MeshletMesh to that slice of the buffer. Then saves the file as a GLB.
  • Added asset processor aliases, so that users can write "MeshletMeshProcessor" in their meta files instead of the very verbose underlying LoadAndSave type path.

TODO

  • Remove the original mesh data from the processed asset to save a significant amount of bytes (can only be done with single-file embedded GLTF or GLB files)
  • Do we need logs in the asset saver? It takes a significant amount of time, and the asset server itself does not seem to log that something is busy processing.
  • Handle other types of meshes better? Should we automatically skip meshes with alpha blended or masked materials which are unsupported by the meshlet feature? Should we automatically remove extra vertex attributes like vertex colors, or should it be an error (MeshletMesh only supports a a pre-defined set of vertex attributes)?
  • Should bevy_gltf meshlet_processor feature imply bevy_asset asset_processor? How should the features be setup? I'm not entirely sure the current setup is correct across bevy_pbr/bevy_gltf/bevy_internal/bevy.
  • Using single-file embedded GLTF's means that images are embedded in the processed file, instead of being separate files that can be run through the asset processor themselves.
  • Performance. There's definitely a lot that could be improved/parallelized, both in the asset saver and in MeshletMesh::from_mesh().

Testing

  • Did you test these changes? If so, how?
    • Tested on bistro and the bunny
  • Are there any parts that need more testing?
    • There's a lot of permutations that need testing:
    • GLTF + separate bin containing mesh data (not recommended, but probably needs testing and maybe a warning in the asset saver)
    • GLTF with embedded bin
    • GLB
    • All the above, with various combinations of meshes (different unsupported vertex attributes, extensions like Blender_bevy_components_workflow, etc)
    • Need to test that bevy_gltf can be compiled individually with the meshlet/meshlet_processor features turned on or off. Feature flags are hard to get right, I might have missed something.
    • Forgetting to add the MeshletPlugin
    • Need to make sure existing examples and asset workflows did not break.
  • How can other people (reviewers) test your changes? Is there anything specific they need to know?
    • Copy the meta file from assets/models/bunny.gltf.meta. Run bevy with features asset_processor,meshlet_processer, and make sure to both add the MeshletPlugin to your app, and set AssetMode::Processed in AssetPlugin. See the meshlet example for more details.
    • If you want to spawn an entire GLTF scene instead of pulling out a single MeshletMesh, spawn a SceneBundle using foo.gltf#Scene0. Same exact API as how it works for non-meshlet mesh GLTFs.

Changelog

  • Added bevy::gltf RawGltf and RawGltfLoader
  • bevy::gltf::GltfLoaderSettings is now marked as #[serde(default)]
  • Added AssetApp and AssetProcessor register_asset_processor_with_alias()

image

@JMS55 JMS55 added A-Assets Load files from disk to use for things like images, models, and sounds A-Scenes Serialized ECS data stored on the disk labels May 20, 2024
@JMS55 JMS55 added this to the 0.15 milestone May 20, 2024
@alice-i-cecile alice-i-cecile added D-Complex Quite challenging from either a design or technical perspective. Ask for help! S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged X-Uncontroversial This work is generally agreed upon labels May 21, 2024
@JMS55 JMS55 modified the milestones: 0.15, 0.14 May 24, 2024
@alice-i-cecile alice-i-cecile added S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged S-Needs-SME Decision or review from an SME is required X-Contentious There are nontrivial implications that should be thought through and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward X-Uncontroversial This work is generally agreed upon labels Jun 3, 2024
@alice-i-cecile alice-i-cecile removed this from the 0.14 milestone Jun 4, 2024
@alice-i-cecile alice-i-cecile removed the S-Needs-SME Decision or review from an SME is required label Jun 4, 2024
@alice-i-cecile
Copy link
Member

I've discussed this PR with Jasmine. Our conclusion is that getting this into a genuinely nice state for 0.14 isn't feasible. Furthermore, I'm worried that this isn't a good candidate for Bevy's first real pre-processing workflow and is going to be much less fraught to tackle after we've nailed down the basics.

github-merge-queue bot pushed a commit that referenced this pull request Jun 18, 2024
This change reworks `find_connected_meshlets` to scale more linearly
with the mesh size, which significantly reduces the cost of building
meshlet representations. As a small extra complexity reduction, it moves
`simplify_scale` call out of the loop so that it's called once (it only
depends on the vertex data => is safe to cache).

The new implementation of connectivity analysis builds edge=>meshlet
list data structure, which allows us to only iterate through
`tuple_combinations` of a (usually) small list. There is still some
redundancy as if two meshlets share two edges, they will be represented
in the meshlet lists twice, but it's overall much faster.

Since the hash traversal is non-deterministic, to keep this part of the
algorithm deterministic for reproducible results we sort the output
adjacency lists.

Overall this reduces the time to process bunny mesh from ~4.2s to ~1.7s
when using release; in unoptimized builds the delta is even more
significant.

This was tested by using #13431
and:

a) comparing the result of `find_connected_meshlets` using old and new
code; they are equal in all steps of the clustering process
b) comparing the rendered result of the old code vs new code *after*
making the rest of the algorithm deterministic: right now the loop that
iterates through the result of `group_meshlets()` call executes in
different order between program runs. This is orthogonal to this change
and can be fixed separately.

Note: a future change can shrink the processing time further from ~1.7s
to ~0.4s with a small diff but that requires an update to meshopt crate
which is pending in gwihlidal/meshopt-rs#42.
This change is independent.
github-merge-queue bot pushed a commit that referenced this pull request Jun 20, 2024
This is a followup to #13904
based on the discussion there, and switches two HashMaps that used
meshlet ids as keys to Vec.

In addition to a small further performance boost for `from_mesh` (1.66s
=> 1.60s), this makes processing deterministic modulo threading issues
wrt CRT rand described in the linked PR. This is valuable for debugging,
as you can visually or programmatically inspect the meshlet distribution
before/after making changes that should not change the output, whereas
previously every asset rebuild would change the meshlet structure.

Tested with #13431; after this
change, the visual output of meshlets is consistent between asset
rebuilds, and the MD5 of the output GLB file does not change either,
which was not the case before.
mockersf pushed a commit that referenced this pull request Jun 27, 2024
This change reworks `find_connected_meshlets` to scale more linearly
with the mesh size, which significantly reduces the cost of building
meshlet representations. As a small extra complexity reduction, it moves
`simplify_scale` call out of the loop so that it's called once (it only
depends on the vertex data => is safe to cache).

The new implementation of connectivity analysis builds edge=>meshlet
list data structure, which allows us to only iterate through
`tuple_combinations` of a (usually) small list. There is still some
redundancy as if two meshlets share two edges, they will be represented
in the meshlet lists twice, but it's overall much faster.

Since the hash traversal is non-deterministic, to keep this part of the
algorithm deterministic for reproducible results we sort the output
adjacency lists.

Overall this reduces the time to process bunny mesh from ~4.2s to ~1.7s
when using release; in unoptimized builds the delta is even more
significant.

This was tested by using #13431
and:

a) comparing the result of `find_connected_meshlets` using old and new
code; they are equal in all steps of the clustering process
b) comparing the rendered result of the old code vs new code *after*
making the rest of the algorithm deterministic: right now the loop that
iterates through the result of `group_meshlets()` call executes in
different order between program runs. This is orthogonal to this change
and can be fixed separately.

Note: a future change can shrink the processing time further from ~1.7s
to ~0.4s with a small diff but that requires an update to meshopt crate
which is pending in gwihlidal/meshopt-rs#42.
This change is independent.
mockersf pushed a commit that referenced this pull request Jun 27, 2024
This is a followup to #13904
based on the discussion there, and switches two HashMaps that used
meshlet ids as keys to Vec.

In addition to a small further performance boost for `from_mesh` (1.66s
=> 1.60s), this makes processing deterministic modulo threading issues
wrt CRT rand described in the linked PR. This is valuable for debugging,
as you can visually or programmatically inspect the meshlet distribution
before/after making changes that should not change the output, whereas
previously every asset rebuild would change the meshlet structure.

Tested with #13431; after this
change, the visual output of meshlets is consistent between asset
rebuilds, and the MD5 of the output GLB file does not change either,
which was not the case before.
@JMS55 JMS55 marked this pull request as draft July 4, 2024 20:04
@JMS55
Copy link
Contributor Author

JMS55 commented Jul 25, 2024

I really need this to be able to test meshlets on real scene, but it's currently blocked on BSN (emitting new GLTF files has proved to be not fun), and probably better asset processing APIs.

@JMS55 JMS55 closed this Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Assets Load files from disk to use for things like images, models, and sounds A-Scenes Serialized ECS data stored on the disk D-Complex Quite challenging from either a design or technical perspective. Ask for help! S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged X-Contentious There are nontrivial implications that should be thought through
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants