Sort encoded Skeleton dictionary for backwards compatibility

[roomrys]

Description

In #1970, we created our own SkeletonEncoder class to move away from using jsonpickle.encode. And, while the SkeletonEncoder paired with the SkeletonDecoder works perfectly fine, we get some strange behavior when trying to load Skeleton jsons across versions of SLEAP (that still use jsonpickle.decode).

The only difference seemed to be the ordering of the keys in the dictionaries; thus, this PR sorts the Skeleton dictionaries before encoding them - which has proven to do the trick with some manual testing across versions.

Unsorted skeleton (breaks backwards compatibility)

{"directed": true, "multigraph": true, "graph": {"name": "Skeleton-0", "num_edges_inserted": 46}, "nodes": [{"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["head", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["thorax", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["abdomen", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingL", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingR", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegL4", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegR4", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegL4", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegR4", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegL4", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegR4", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeL", 1.0]}}}, {"id": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeR", 1.0]}}}], "links": [{"edge_insert_idx": 44, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}, "source": {"py/id": 1}, "target": {"py/id": 12}, "key": 0}, {"edge_insert_idx": 45, "type": {"py/id": 14}, "source": {"py/id": 1}, "target": {"py/id": 13}, "key": 0}, {"edge_insert_idx": 34, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 1}, "key": 0}, {"edge_insert_idx": 35, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 3}, "key": 0}, {"edge_insert_idx": 36, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 4}, "key": 0}, {"edge_insert_idx": 37, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 5}, "key": 0}, {"edge_insert_idx": 38, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 6}, "key": 0}, {"edge_insert_idx": 39, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 7}, "key": 0}, {"edge_insert_idx": 40, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 8}, "key": 0}, {"edge_insert_idx": 41, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 9}, "key": 0}, {"edge_insert_idx": 42, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 10}, "key": 0}, {"edge_insert_idx": 43, "type": {"py/id": 14}, "source": {"py/id": 2}, "target": {"py/id": 11}, "key": 0}]}

Sorted skeleton (backwards compatible)

{"directed": true, "graph": {"name": "Skeleton-0", "num_edges_inserted": 46}, "links": [{"edge_insert_idx": 44, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["head", 1.0]}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeL", 1.0]}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 45, "key": 0, "source": {"py/id": 1}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeR", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 34, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["thorax", 1.0]}}, "target": {"py/id": 1}, "type": {"py/id": 3}}, {"edge_insert_idx": 35, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["abdomen", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 36, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingL", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 37, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingR", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 38, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 39, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegR4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 40, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 41, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegR4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 42, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 43, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegR4", 1.0]}}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}, {"id": {"py/id": 7}}, {"id": {"py/id": 8}}, {"id": {"py/id": 9}}, {"id": {"py/id": 10}}, {"id": {"py/id": 11}}, {"id": {"py/id": 12}}, {"id": {"py/id": 13}}, {"id": {"py/id": 14}}, {"id": {"py/id": 2}}, {"id": {"py/id": 4}}]}

v1.3.3 Skeleton

{"directed": true, "graph": {"name": "Skeleton-0", "num_edges_inserted": 46}, "links": [{"edge_insert_idx": 44, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["head", 1.0]}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeL", 1.0]}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 45, "key": 0, "source": {"py/id": 1}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeR", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 34, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["thorax", 1.0]}}, "target": {"py/id": 1}, "type": {"py/id": 3}}, {"edge_insert_idx": 35, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["abdomen", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 36, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingL", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 37, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingR", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 38, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 39, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegR4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 40, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 41, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegR4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 42, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 43, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegR4", 1.0]}}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}, {"id": {"py/id": 7}}, {"id": {"py/id": 8}}, {"id": {"py/id": 9}}, {"id": {"py/id": 10}}, {"id": {"py/id": 11}}, {"id": {"py/id": 12}}, {"id": {"py/id": 13}}, {"id": {"py/id": 14}}, {"id": {"py/id": 2}}, {"id": {"py/id": 4}}]}

Types of changes

• Bugfix
• New feature
• Refactor / Code style update (no logical changes)
• Build / CI changes
• Documentation Update
• Other (explain)

Does this address any currently open issues?

• Bump to 1.4.1a3 #1972
• Handle skeleton encoding internally #1970
• Handle skeleton decoding internally #1961
• Update python versions and dependencies #1841

Outside contributors checklist

• Review the guidelines for contributing to this repository
• Read and sign the CLA and add yourself to the authors list
• Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
• Add tests that prove your fix is effective or that your feature works
• Add necessary documentation (if appropriate)

Thank you for contributing to SLEAP!

❤️

Summary by CodeRabbit

• New Features
  • Introduced a method to sort dictionary keys recursively during the encoding process for improved data consistency.
• Bug Fixes
  • Enhanced the encoding process to ensure compatibility with earlier versions of the SLEAP framework.
• Tests
  • Added assertions to verify that encoded JSON keys are sorted, improving robustness and compatibility.

[coderabbitai]

Walkthrough

The changes in this pull request involve modifications to the SkeletonEncoder class in the sleap/skeleton.py file. A new static method, _recursively_sort_dict, is introduced to sort dictionaries recursively by their keys, which is called within the encode method to ensure sorted data for backward compatibility with earlier SLEAP versions. Additionally, a test in tests/test_skeleton.py is updated to verify that the keys in the encoded JSON string are sorted, enhancing the robustness of the encoding process.

Changes

File	Change Summary
sleap/skeleton.py	- Added method: _recursively_sort_dict(dictionary: Dict[str, Any]) in SkeletonEncoder. - Updated method: encode(cls, data: Dict[str, Any]) to include sorting step. - Removed unnecessary line for setting jsonpickle options.
tests/test_skeleton.py	- Updated test_decoded_encoded_Skeleton to assert that keys in the encoded JSON are sorted, including nested dictionaries and lists.

Possibly related PRs

• Handle skeleton encoding internally #1970: The introduction of the SkeletonEncoder class in this PR directly relates to the modifications made to the SkeletonEncoder class in the main PR, as both involve changes to the encoding process within sleap/skeleton.py.

Suggested reviewers

• talmo

Poem

In the land where data hops,
A rabbit sorts and never stops.
With keys aligned in perfect rows,
Compatibility is how it grows!
Encoding magic, swift and bright,
In SLEAP's world, all feels just right! 🐇✨

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.48%. Comparing base (7ed1229) to head (3fcff25).
Report is 55 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1975      +/-   ##
===========================================
+ Coverage    73.30%   75.48%   +2.17%     
===========================================
  Files          134      133       -1     
  Lines        24087    24635     +548     
===========================================
+ Hits         17658    18596     +938     
+ Misses        6429     6039     -390     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[coderabbitai]

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (3)

tests/test_skeleton.py (1)

48-59: LGTM! Comprehensive check for sorted keys enhances backwards compatibility.

The added assertions effectively ensure that the encoded JSON has keys in sorted order, which aligns well with the PR objective of maintaining backwards compatibility. The implementation is thorough, covering nested dictionaries and lists.

Consider renaming the unused loop variable key to _ to indicate it's intentionally unused:

-for key, value in encoded_dict.items():
+for _, value in encoded_dict.items():

Would you like assistance in generating additional test cases to cover edge cases or specific backwards compatibility scenarios?

🧰 Tools

🪛 Ruff

52-52: Loop control variable key not used within loop body

Rename unused key to _key

(B007)

sleap/skeleton.py (2)

423-428: LGTM! Consider adding a version check for optimization.

The addition of sorting the input data addresses the backwards compatibility issue with SLEAP <=1.3.4 as mentioned in the PR objectives. This change is well-implemented and commented.

For potential optimization, you might consider adding a version check to skip the sorting step for newer SLEAP versions. This could look like:

if sleap_version <= Version("1.3.4"):
    sorted_data = cls._recursively_sort_dict(data)
else:
    sorted_data = data

This would maintain backwards compatibility while potentially improving performance for newer versions.

---

432-446: LGTM! Consider adding a type check for potential optimization.

The _recursively_sort_dict method is well-implemented and correctly handles nested structures. It effectively addresses the need for consistent ordering in the encoded data, which is crucial for backwards compatibility.

For a potential optimization, you might consider adding a type check before recursing into list items. This could prevent unnecessary function calls for lists that don't contain dictionaries:

elif isinstance(value, list):
    if any(isinstance(item, dict) for item in value):
        for i, item in enumerate(value):
            if isinstance(item, dict):
                sorted_dict[key][i] = SkeletonEncoder._recursively_sort_dict(item)

This change would skip the recursion for lists that don't contain any dictionaries, potentially improving performance for certain data structures.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 10aae76 and 3fcff25.

📒 Files selected for processing (2)

• sleap/skeleton.py (2 hunks)
• tests/test_skeleton.py (1 hunks)

🧰 Additional context used

🪛 Ruff

tests/test_skeleton.py

52-52: Loop control variable key not used within loop body

Rename unused key to _key

(B007)

🔇 Additional comments (1)

sleap/skeleton.py (1)

423-446: Summary: Changes effectively address backwards compatibility.

The modifications to the SkeletonEncoder class, particularly the addition of the _recursively_sort_dict method and its use in the encode method, successfully address the backwards compatibility issue with SLEAP versions <=1.3.4. The implementation is clean, well-commented, and maintains the overall structure of the class.

These changes align well with the PR objectives, solving the issue of loading Skeleton JSONs across different versions of the SLEAP framework. The consistent ordering of dictionary keys ensures that the encoded data remains compatible with older versions that rely on jsonpickle.decode.

The implementation is solid, and with the suggested minor optimizations, it should provide both backwards compatibility and good performance.

[gitttt-1234]

Subgroup approves