Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema Downgrading System #1387

Merged
merged 119 commits into from
Sep 12, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
119 commits
Select commit Hold shift + click to select a range
11cdc38
refactor versioning tests into their own class
ssteinbach Aug 1, 2022
5cb6143
DRY cleanup in the serializer before other stuff
ssteinbach Aug 12, 2022
c52aede
DRY reduction in the json FILE serializer
ssteinbach Aug 12, 2022
9151fc0
Add io_perf_test to repo
ssteinbach Aug 14, 2022
05d5b4a
Add a call w/ downgrade manifest to io_perf_test
ssteinbach Aug 17, 2022
df297aa
add anydictionary convienence functions
ssteinbach Aug 17, 2022
426c128
Checkpoint broken Writer approach
ssteinbach Aug 17, 2022
5eaf20a
add .cache to gitignore
ssteinbach Aug 22, 2022
81582a3
Test runs, @TODO: check for correctness
ssteinbach Aug 25, 2022
9848c84
cleanup to the test
ssteinbach Aug 25, 2022
440b5dd
formatting tweaks
ssteinbach Aug 25, 2022
3193822
turn on all the tests
ssteinbach Aug 25, 2022
2f9e77d
remove dead code
ssteinbach Aug 25, 2022
33f698e
tightening the AnyDictionary code up a bit
ssteinbach Aug 25, 2022
769cbc3
more odds and ends
ssteinbach Aug 25, 2022
dd128b0
add override tags
ssteinbach Aug 25, 2022
5ac78c9
noodling
ssteinbach Aug 25, 2022
eaf2801
refactored into function
ssteinbach Aug 25, 2022
991febb
only conver to AnyDictionary if downgrade detected
ssteinbach Aug 25, 2022
e92389c
add perf tests for no-downgrade scenarios
ssteinbach Aug 25, 2022
28f5669
use const auto& where possible in range-for
ssteinbach Aug 25, 2022
97cce91
in this case, faster to std::string concatenation
ssteinbach Aug 25, 2022
34434c1
checkpointing work with all tests passing
ssteinbach Aug 25, 2022
6b2842c
check point design before pushing arguments through
ssteinbach Aug 25, 2022
5eb7ed1
fix a test bug
ssteinbach Aug 26, 2022
e3cab5b
plumb target_family_label_spec through
ssteinbach Aug 26, 2022
67c0412
pass target_family_label_spec into io_perf_test
ssteinbach Aug 26, 2022
d45f6bd
add downgrade version family sets (name is cbb)
ssteinbach Aug 27, 2022
1c857ce
add some error checking on inserting families
ssteinbach Aug 27, 2022
0b0ec22
minor cleanup of dead comments and stuff
ssteinbach Aug 27, 2022
a221c55
clean up io_perf_test
ssteinbach Aug 30, 2022
5a8ab28
autogen version info struct
ssteinbach Aug 31, 2022
95075df
tweak the io_perf_test to downgrade again
ssteinbach Aug 31, 2022
e814381
add exceptions for overwriting up/downgrade fn
ssteinbach Aug 31, 2022
f7c7aa2
add exception when double registering a type
ssteinbach Aug 31, 2022
f0d148d
move schema version types into typeRegistry
ssteinbach Aug 31, 2022
f097827
remove "family" level from internal maps
ssteinbach Aug 31, 2022
ad41fd6
mark the internal map as const
ssteinbach Aug 31, 2022
7fc01e1
move FAMILY_VERSION_MAP file to CORE_VERSION_MAP
ssteinbach Aug 31, 2022
e0cf66c
update autogen_version_info to match new files
ssteinbach Aug 31, 2022
36e8bf1
Autogen_version_info -> CORE_VERSION_MAP term
ssteinbach Sep 1, 2022
f0a7871
build out CORE_VERSION_MAP.last.cpp
ssteinbach Sep 1, 2022
1d01b4d
type registry allows mutliple-regsiter if same
ssteinbach Sep 1, 2022
feddbda
plumb through bindings (schema_version_map)
ssteinbach Sep 1, 2022
47501ec
truing more names up in bindings
ssteinbach Sep 1, 2022
1078a8d
update tests for CORE_VERSION_MAP generation
ssteinbach Sep 1, 2022
1fa6542
back out of double-register exception
ssteinbach Sep 1, 2022
342f207
add version manifest plugin
ssteinbach Sep 1, 2022
3726e4f
add version manifest plugin testing
ssteinbach Sep 2, 2022
82b2781
lint pass
ssteinbach Sep 2, 2022
dd26ded
comment formatting for RTD
ssteinbach Sep 2, 2022
85f2250
lint
ssteinbach Sep 2, 2022
ac0c24f
lint again
ssteinbach Sep 2, 2022
b1707bc
linty lint lint
ssteinbach Sep 2, 2022
75fb9c8
update the doc model
ssteinbach Sep 2, 2022
103679d
update docs for better RTD readability
ssteinbach Sep 2, 2022
0743f12
bump plugin docs
ssteinbach Sep 2, 2022
16cf426
force sort of autogen version map
ssteinbach Sep 2, 2022
2b93376
force unix line endings for the version map test
ssteinbach Sep 2, 2022
b5584f1
fix manifest layering for the version_manifests
ssteinbach Sep 4, 2022
34d33a6
eval->json.loads
ssteinbach Sep 4, 2022
5de438c
fix name of the downgrade_function_from decorator
ssteinbach Sep 4, 2022
457e9e9
add docs about new downgrading schemas
ssteinbach Sep 5, 2022
8aeffc3
remove dead code
ssteinbach Sep 5, 2022
82f15c5
unordered_map in serializableObject instead of map
ssteinbach Sep 5, 2022
5712202
all io_perf_test on by default
ssteinbach Sep 5, 2022
90fdac1
add upgrade_downgrade_example in C++
ssteinbach Sep 5, 2022
dcef19b
Add notes to environment variables markdown.
ssteinbach Sep 5, 2022
d29ab26
typo
ssteinbach Sep 5, 2022
5ab1e67
typo
ssteinbach Sep 5, 2022
0c680bb
Improve error handling and text for env var errors
ssteinbach Sep 6, 2022
5f22bb5
various tightening, notes, and documentation
ssteinbach Sep 6, 2022
88de08a
Simplify docs
ssteinbach Sep 6, 2022
3038c77
update plugin autodocs
ssteinbach Sep 6, 2022
110251c
update docs more
ssteinbach Sep 6, 2022
1f45885
update docs to quiet warnings
ssteinbach Sep 6, 2022
2b9c779
remove dead code and use type alias
ssteinbach Sep 6, 2022
3b197a2
remove dead code
ssteinbach Sep 6, 2022
cc1112a
remove unhelpful comment
ssteinbach Sep 6, 2022
d8ebf23
don't erase the enabled flag per our policy
ssteinbach Sep 6, 2022
ed5aae7
remove breadcrumbing from the downgrade function
ssteinbach Sep 6, 2022
1f77404
optional<schema_version_map*> -> pointer
ssteinbach Sep 6, 2022
93cae2a
AnyDictionary::get_default -> get_if_set
ssteinbach Sep 6, 2022
1dc926b
mark AnyDictionary methods const
ssteinbach Sep 6, 2022
53399ef
remove dead code
ssteinbach Sep 7, 2022
0ecc418
create a temp directory for io_perf_test
ssteinbach Sep 7, 2022
040eecc
review note
ssteinbach Sep 7, 2022
887b02c
remove printout of tmp dir path if it is removed
ssteinbach Sep 7, 2022
81d8c65
clean up imports
ssteinbach Sep 7, 2022
e3aa7a3
stray const
ssteinbach Sep 7, 2022
b9b5d11
sanitize line endings in serialized versions test
ssteinbach Sep 7, 2022
13b11b6
remove empty lines from comparison in version test
ssteinbach Sep 7, 2022
96ccfa0
lint
ssteinbach Sep 8, 2022
f268dcc
typo
ssteinbach Sep 8, 2022
d6bb4b5
comment out debug code
ssteinbach Sep 8, 2022
0633af1
add license headers
ssteinbach Sep 8, 2022
0ff7036
link to file for examples
ssteinbach Sep 8, 2022
f9696b4
spell pass
ssteinbach Sep 8, 2022
68310ca
remove dead code from core
ssteinbach Sep 8, 2022
68a0263
spell fix
ssteinbach Sep 8, 2022
5d1e78a
add license header
ssteinbach Sep 8, 2022
1349dd7
doc tuning
ssteinbach Sep 8, 2022
ea353dd
hiding links that won't exist until PR is landed
ssteinbach Sep 8, 2022
6685d41
updating docs
ssteinbach Sep 8, 2022
4b7f93f
Add docs to core/__init__.py
ssteinbach Sep 8, 2022
91cfd7a
add docstrings to core module
ssteinbach Sep 8, 2022
02d4b95
lint
ssteinbach Sep 8, 2022
05cadb2
Removed need for root() method and use any/swap
ssteinbach Sep 8, 2022
58f6418
checkpoint perf 1.66s
ssteinbach Sep 8, 2022
e30d36f
add replace_back function (further work needed)
ssteinbach Sep 8, 2022
37ae0c3
io_perf_test asserts
ssteinbach Sep 9, 2022
efee648
use insert directly for up/downgrade functions
ssteinbach Sep 9, 2022
b49033c
Performance tuning
ssteinbach Sep 9, 2022
3e00dad
add nullptr check and compare type hash code
ssteinbach Sep 9, 2022
cf5a5ef
print ratio of duration in the io_perf_test
ssteinbach Sep 9, 2022
d97241b
use std::chrono... to get a float seconds count
ssteinbach Sep 9, 2022
381a762
one more string copy prevented
ssteinbach Sep 9, 2022
83c9979
remove double exception checks
ssteinbach Sep 9, 2022
026c8f0
Remove some unneeded imports.
ssteinbach Sep 9, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ htmlcov
.vscode/
xcuserdata/
.venv/
.cache

# Pycharm metadata
.idea/
Expand Down
7 changes: 7 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,13 @@ doc-plugins:
doc-plugins-update:
@python src/py-opentimelineio/opentimelineio/console/autogen_plugin_documentation.py -o docs/tutorials/otio-plugins.md --public-only --sanitized-paths

# build the CORE_VERSION_MAP cpp file
version-map:
@python src/py-opentimelineio/opentimelineio/console/autogen_version_map.py -i src/opentimelineio/CORE_VERSION_MAP.last.cpp --dryrun

version-map-update:
@python src/py-opentimelineio/opentimelineio/console/autogen_version_map.py -i src/opentimelineio/CORE_VERSION_MAP.last.cpp -o src/opentimelineio/CORE_VERSION_MAP.cpp

# generate documentation in html
doc-html:
@# if you just want to build the docs yourself outside of RTD
Expand Down
3 changes: 2 additions & 1 deletion docs/tutorials/otio-env-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,12 @@ various aspects of OTIO.

## Plugin Configuration

These variables must be set _before_ the OpenTimelineIO python library is imported.
These variables must be set _before_ the OpenTimelineIO python library is imported. They only impact the python library. The C++ library has no environment variables.

- `OTIO_PLUGIN_MANIFEST_PATH`: a ":" separated string with paths to .manifest.json files that contain OTIO plugin manifests. See: [Tutorial on how to write an adapter plugin](write-an-adapter).
- `OTIO_DEFAULT_MEDIA_LINKER`: the name of the default media linker to use after reading a file, if "" then no media linker is automatically invoked.
- `OTIO_DISABLE_PKG_RESOURCE_PLUGINS`: By default, OTIO will use the pkg_resource entry_points mechanism to discover plugins that have been installed into the current python environment. pkg_resources, however, can be slow in certain cases, so for users who wish to disable this behavior, this variable can be set to 1.
- `OTIO_DEFAULT_TARGET_VERSION_FAMILY_LABEL`: if no downgrade arguments are passed to `write_to_file`/`write_to_string`, use the downgrade manifest specified by the family/label combination in the variable. Variable is of the form FAMILY:LABEL. Only one tuple of FAMILY:LABEL may be specified.

## Unit tests

Expand Down
22 changes: 21 additions & 1 deletion docs/tutorials/otio-plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ OpenTimelineIO Final Cut Pro 7 XML Adapter.
### otio_json

```
This adapter lets you read and write native .otio files
Adapter for reading and writing native .otio json files.
```

*source*: `opentimelineio/adapters/otio_json.py`
Expand Down Expand Up @@ -147,14 +147,23 @@ Serializes an OpenTimelineIO object into a file
indent (int): number of spaces for each json indentation level.
Use -1 for no indentation or newlines.

If target_schema_versions is None and the environment variable
"OTIO_DEFAULT_TARGET_VERSION_FAMILY_LABEL" is set, will read a map out of
that for downgrade target. The variable should be of the form
FAMILY:LABEL, for example "MYSTUDIO:JUNE2022".

Returns:
bool: Write success

Raises:
ValueError: on write error
otio.exceptions.InvalidEnvironmentVariableError: if there is a problem
with the default environment variable
"OTIO_DEFAULT_TARGET_VERSION_FAMILY_LABEL".
```
- input_otio
- filepath
- target_schema_versions
- indent
- write_to_string:
```
Expand All @@ -165,10 +174,21 @@ Serializes an OpenTimelineIO object into a string
indent (int): number of spaces for each json indentation level. Use
-1 for no indentation or newlines.

If target_schema_versions is None and the environment variable
"OTIO_DEFAULT_TARGET_VERSION_FAMILY_LABEL" is set, will read a map out of
that for downgrade target. The variable should be of the form
FAMILY:LABEL, for example "MYSTUDIO:JUNE2022".

Returns:
str: A json serialized string representation

Raises:
otio.exceptions.InvalidEnvironmentVariableError: if there is a problem
with the default environment variable
"OTIO_DEFAULT_TARGET_VERSION_FAMILY_LABEL".
```
- input_otio
- target_schema_versions
- indent


Expand Down
1 change: 1 addition & 0 deletions docs/tutorials/otio-serialized-schema-only-fields.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ parameters:
- *hooks*
- *media_linkers*
- *schemadefs*
- *version_manifests*

### SerializableObject.1

Expand Down
1 change: 1 addition & 0 deletions docs/tutorials/otio-serialized-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,7 @@ parameters:
- *hooks*: Hooks that hooks scripts can be attached to.
- *media_linkers*: Media Linkers this manifest describes.
- *schemadefs*: Schemadefs this manifest describes.
- *version_manifests*: Sets of versions to downgrade schemas to.

### SerializableObject.1

Expand Down
Loading