-
Notifications
You must be signed in to change notification settings - Fork 903
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DataCatalog2.0]: Draft of AbstractDataCatalog
and KedroDataCatalog
(work in progress)
#4070
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
7 tasks
Signed-off-by: Elena Khaustova <[email protected]>
ElenaKhaustova
changed the title
[DataCatalog2.0]: Refactor dataset factory resolution logic (work in progress)
[DataCatalog2.0]: Draft of Aug 7, 2024
AbstractDataCatalog
and KedroDataCatalog
(work in progress)
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Signed-off-by: Elena Khaustova <[email protected]>
Replaces with #4123 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Solves #3925, #3926, #3916
Development notes
This PR includes a draft of the following:
AbstractDataCatalog
andKedroDataCatalog(AbstractDataCatalog)
AbstractDataCatalog
supports instantiation from configuration and/or datasetsAbstractDataCatalog
stores the configuration provided_get_dataset()
toresolve_patterns()
_dataset_patterns
and_default_patterns
now obtained from config at the__init__
resolved_ds_configs
property to store resolved datasets' configurations at the catalog leveladd()
method adds or replaces the dataset and its configurationadd_feed_dict()
renamed toadd_from_dict()
_runtime_patterns
catalog field to keep the logic of processing dataset/default/runtime patterns clearshallow_copy()
method used to add extra_dataset_patterns at runtime, replaced it with dedicated -add_runtime_patterns()
method_FrozenDatasets
and access datasets as propertiesadd(replace=True)
KedroDataCatalog
mutable:datasets
property public not to encourage behaviour when users configure the catalog via modifying thedatasets
dictionary_datasets
property remained protected, but publicdatasets
property was added, returning a deep copy of_datasets
while the setter is still not allowed; the same is applied to the_resolved_ds_configs
property_datasets
via thecatalog.add()
methodAbstractDataCatalog
compatible with the current runners' implementation several methods -release()
,confirm()
andexists()
were kept as the part of interface. But they only have a meaningful implementation forKedroDataCatalog
Developer Certificate of Origin
We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a
Signed-off-by
line in the commit message. See our wiki for guidance.If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.
Developer Certificate of Origin
We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a
Signed-off-by
line in the commit message. See our wiki for guidance.If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.
Checklist
RELEASE.md
file