vertexcodec: Reintroduce tracing and add roundtrip fuzzing #780
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In addition to validating all decoders for memory safety we can also validate that the vertex encoder in particular round-trips safely. Index encoder may rotate triangles so it's not a perfect binary match, and index sequence encoder is rarely used and very simple, but vertex encoder is more complicated and may become more so with v1.
This is a partial revert of 4600e89 which removed tracing support from both codecs; to make further progress on vertex codec v1 we need some amount of tracing back, and while the tracing we used to have isn't sufficient it's a good start.
For now we just need a single trace level that incorporates both the macro stats (per byte) and the micro stats (within the individual byte). Instead of separately collecting occurrences/sizes for bit groups we just record the size and print all numbers as percentages within the given group.
Within each byte group, we count the number of consistent bits: bits that, in that byte group, all match the corresponding bit from the last vertex. Consistent bits amplify compression rates when they are properly aligned. Note that this is not exactly the same as having small deltas, eg 0xff vs 0x00 is still consistent from the delta perspective in most high bits, but it would be the same if we used xor deltas (which may be an option worth having). This is also *not* the same as having bits consistent between consecutive bytes: in a very random sequence, about half of the bits in each position usually match from one value to the next, but that's a very low consistency as it's hard to take advantage of.
bitg[0] is always 0 because we accumulate only sizes, and we store zero data bits for bit0... we might need to print count percentages in the future instead but this is probably ok for now. Also reduce the space wasted by extra whitespace.
|
Example trace, 24 byte vertex, 12 byte float3 position, two ushort2 UV channels, packed normal (1-11-10-10):
|
Tabs only give us a limited amount of alignment if some byte streams are much longer than others; instead, explicitly specify the width for all numbers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change reintroduces TRACE macro to vertex encoder; we used to have tracing code but it was removed in 2020 as the v0 codec was finalized. However, in order to make progress towards v1 codec we need to bring some of it back; this change adds byte tracking back with a more concise but complete summary display, and also adds bit consistency tracking (a bit is considered consistent if it is the same as the last bit in the same position for the entire byte group). Bit consistency tracking helps analyze opportunities where rotation or reorganization of data can improve compression.
Note that tracing relies on global state and is not thread-safe; since it prints data to stdout it's not really useful in a threaded context either way, and the thread safety issues should not cause any runtime issues, just incorrect data aggregation (nor will tracing ever be enabled by default or exposed as an official configuration, so it's mostly moot).
Also add roundtrip fuzzing to the existing codec fuzzer; if we were to implement v1, we would need a way to ensure data is compressed losslessly.
This contribution is sponsored by Valve.