Add migration steps for TemplatedConfigLoader to OmegaConfigLoader
#2904
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Jo Stichbury <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
merelcht
force-pushed
the
migration-docs-tcl
branch
from
f902f45
to
47f22f5
Compare
2 tasks
# Conflicts: # docs/source/configuration/config_loader_migration.md
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
merelcht
commented
Aug 18, 2023
merelcht
requested review from
ankatiyar
and removed request for
astrojuanlu,
noklam and
yetudada
datajoely
reviewed
Aug 18, 2023
| boats: | ||
| users: | ||
| - fred | ||
| - - "${write_only_user|ron}" |
There was a problem hiding this comment.
you may want to define this as it currently doesn't have context
|
|
||
| ### 8. Jinja2 | ||
| `OmegaConfigLoader` does not support Jinja2 syntax in configuration. However, users can achieve similar functionality with the `OmegaConfigLoader` in combination with [dataset factories](../data/data_catalog.md#load-multiple-datasets-with-similar-configuration-using-dataset-factories). | ||
| If you take the example from [the `TemplatedConfigLoader` with Jinja2 documentation](advanced_configuration.md#how-to-use-jinja2-syntax-in-configuration) you can rewrite your configuration as follows to work with `OmegaConfigLoader`: |
There was a problem hiding this comment.
I wonder if we want to provide some opinion here on why Jinja is a suboptimal solution here, not built for a whitespaced language etc.
Or point to articles like this one (last paragraph of "Pitfall 1")
Co-authored-by: Joel <[email protected]>
2 tasks
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| ``` | ||
|
|
||
| ### 4. File Format Support | ||
| `OmegaConfigLoader` supports only `yaml` and `json` file formats. Make sure that all your configuration files are in one of these formats. If you were using other formats with `TemplatedConfigLoader`, convert them to `yaml` or `json`. |
There was a problem hiding this comment.
[Kedro.weaselwords] 'only' is a weasel word!
| ``` | ||
|
|
||
| ### 8. Deprecation of Jinja2 | ||
| `OmegaConfigLoader` does not support Jinja2 syntax in configuration. However, users can achieve similar functionality with the `OmegaConfigLoader` in combination with [dataset factories](../data/data_catalog.md#load-multiple-datasets-with-similar-configuration-using-dataset-factories). |
There was a problem hiding this comment.
[Kedro.toowordy] 'However' is too wordy
merelcht
commented
Aug 29, 2023
Signed-off-by: Merel Theisen <[email protected]>
|
@stichbury I tried resolving some of the Vale comments, but I wasn't too succesful 😅 Would you mind reviewing? |
|
@stichbury is on PTO, I suspect the bogus "sentence-style capitalization" warnings come because of the numbers 🤔 Other than that, you're capitalizing only the first word of the header, which as far as I know is the intended style. Maybe we can ignore those for now? |
ankatiyar
reviewed
Aug 29, 2023
|
|
||
| In this example, the `"catalog"` key points to the default catalog patterns specified in the `OmegaConfigLoader` class. | ||
|
|
||
| ### 6. Templating of values |
There was a problem hiding this comment.
I think, under this section, it would also be helpful to flag that the variable interpolation is only scoped to a particular configuration and the same environment. For eg., templated keys from local will not overwrite base and the resolution of the keys happens within the environment which is different from how TemplatedConfigLoader uses the globals.
There was a problem hiding this comment.
Second on this. Do we already have the relevant docs in OmegaConfigLoader's page? Obviously just move everything to global would work(this is how template config loader works), but we may want to advise to use the catalog_globals.yml etc to replace the use of globals.py for certain things.
noklam
reviewed
Aug 29, 2023
Comment on lines
+188
to
+189
| You no longer need to set `CONFIG_LOADER_ARGS` variable in [`src/<package_name>/settings.py`](../kedro_project_setup/settings.md) to find this `globals.yml` file, because the | ||
| `OmegaConfigLoader` is configured to pick up files named `globals.yml` by default. |
There was a problem hiding this comment.
Should we add a link in case they name it something other than globals.yml?
There was a problem hiding this comment.
What's the replacement for globals_dict?
There was a problem hiding this comment.
There's no replacement. That's why I wrote "The OmegaConfigLoader requires global values to be provided in a globals.yml file.", but I can explicitly mention that the globals_dict is not supported with OCL for clarity.
There was a problem hiding this comment.
"Should we add a link in case they name it something other than globals.yml?" @noklam what should we link to?
There was a problem hiding this comment.
@merelcht How about https://docs.kedro.org/en/latest/configuration/advanced_configuration.html#how-to-use-global-variables-with-the-omegaconfigloader?
Btw I think we should bubble up OmegaConfigLoader docs since we are trying to make it default. The hierarchy is also broken there is nothing inside OmegaConfigLoader. Cc @stichbury @ankatiyar
There was a problem hiding this comment.
The hierarchy is also broken there is nothing inside OmegaConfigLoader.
I don't think it's broken, we don't have any sub-sections under OmegaConfigLoader because those are all under "Advanced Kedro configuration". We should probably clean this up when all docs on ConfigLoader and TemplatedConfigLoader are removed as part of: #2692
Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Merel Theisen <[email protected]>
The RTD is passing again 🙂 |
|
Unrelated but I see that the Vale reviewer action keeps doing a whole round of reviews after every push. I'll try to see if there's a way for it to be less annoying 😅 |
6 tasks
18 tasks
astrojuanlu
approved these changes
Aug 30, 2023
There was a problem hiding this comment.
This is fantastic ⭐ Thanks a lot @merelcht and reviewers! Left a couple of comments, other than that let's get this merged 
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
ankatiyar
added a commit
that referenced
this pull request
Sep 12, 2023
* Remove non-user-facing release notes about pip pin (#2854) Signed-off-by: Jason Hite <[email protected]> * Allow for customization of pipeline templates Controlled via `settings.PIPELINE_TEMPLATE_PATH`. Signed-off-by: Jason Hite <[email protected]> * Update lookup based on feedback Instead of lookups on `settings.py` just statically look for a `templates` directory in the project root. If this folder is present it will override the default global template, otherwise fall back to the previous global default. Signed-off-by: Jason Hite <[email protected]> * Add CLI flag for pipeline creation template -t/--template Precedence for template_path is: command line > project templates/pipeline dir > global default pipeline create now will also print the template folder used. Signed-off-by: Jason Hite <[email protected]> * First basic unit test TODO is testing the CLI flag too, and maybe testing that the cleanup works with custom templates. Signed-off-by: Jason Hite <[email protected]> * Add test to check overriding template path on the CLI Signed-off-by: Jason Hite <[email protected]> * Linter formatting fixes Signed-off-by: Jason Hite <[email protected]> * Release 0.18.12 (#2871) * bump version and prepare for new release Signed-off-by: SajidAlamQB <[email protected]> * Update RELEASE.md Added community contributions * add DeltaTableDataSet to rst Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> * temporarily use the fixed kedro-plugins branch Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * test docsbuild on merels branch Signed-off-by: SajidAlamQB <[email protected]> * use kedro-datasets new release Signed-off-by: SajidAlamQB <[email protected]> --------- Signed-off-by: SajidAlamQB <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Fix broken link to Prefect website in deploy guide (#2885) Signed-off-by: Jason Hite <[email protected]> * Clarify the <micropkg_name> argument to kedro micropkg package (#2835) Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Allow registering of custom resolvers to `OmegaConfigLoader` (#2869) * Allow registering of custom resolvers to OCL Signed-off-by: Ankita Katiyar <[email protected]> * Complete doc string Signed-off-by: Ankita Katiyar <[email protected]> * Add test for overwritten resolvers Signed-off-by: Ankita Katiyar <[email protected]> * Update test for overwritten resolvers Signed-off-by: Ankita Katiyar <[email protected]> * Remove replace=True by default Signed-off-by: Ankita Katiyar <[email protected]> * Update release notes Signed-off-by: Ankita Katiyar <[email protected]> * Update release notes Signed-off-by: Ankita Katiyar <[email protected]> * Add debug level log for registering new resolver Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Document the use of custom resolvers with `OmegaConfigLoader` (#2896) * Update docs for custom resolvers for OCL Signed-off-by: Ankita Katiyar <[email protected]> * Update FAQ with correct link Signed-off-by: Ankita Katiyar <[email protected]> * Fix typo Co-authored-by: Merel Theisen <[email protected]> * Add note for oc.env Signed-off-by: Ankita Katiyar <[email protected]> * Suggestions from code review Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update kedro pipeline create to use new /conf file structure (#2856) * add parameters_pipelineNane.yml template in the conf folder Signed-off-by: Dmitry Sorokin <[email protected]> * del yml template from parameters folder Signed-off-by: Dmitry Sorokin <[email protected]> * modify delete pipeline command to use correct path to yml files Signed-off-by: Dmitry Sorokin <[email protected]> * modify test_pipeline move parameters foldername to the yml filename Signed-off-by: Dmitry Sorokin <[email protected]> * collect configs to package not only from parameters folder, but from core conf folder also Signed-off-by: Dmitry Sorokin <[email protected]> * move parameters foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <[email protected]> * move parameters foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <[email protected]> * Add changes to RELEASE.md Signed-off-by: Dmitry Sorokin <[email protected]> * Update RELEASE.md Co-authored-by: Merel Theisen <[email protected]> * Update RELEASE.md, change to release 0.18.13 Signed-off-by: Dmitry Sorokin <[email protected]> * save old structure for 'pipeline delete' command for backward compatibility Signed-off-by: Dmitry Sorokin <[email protected]> * modify test_pipeline for 'pipeline delete' backward compability Signed-off-by: Dmitry Sorokin <[email protected]> --------- Signed-off-by: Dmitry Sorokin <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update kedro catalog create to use new /conf structure (#2884) * modify test_catalog, remove nesting in the catalog folder Signed-off-by: Dmitry Sorokin <[email protected]> * move catalog foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <[email protected]> * Update RELEASE.md Signed-off-by: Dmitry Sorokin <[email protected]> * Modify docs, fix broken link Signed-off-by: Dmitry Sorokin <[email protected]> --------- Signed-off-by: Dmitry Sorokin <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add migration steps for `ConfigLoader` to `OmegaConfigLoader` (#2887) * Add migration steps for CL to OCL Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Fix #2498 Adding logging issue 2498 (#2842) * Fixing typo n on line 114 file /docs/source/nodes_and_pipelines/run_a_pipeline.md from issue #2607 Signed-off-by: laizaparizotto <[email protected]> * Removed redundant packages at lines 66, 67 and 68 on file kedro/docs/source/nodes_and_pipelines/run_a_pipeline.md, packages: from kedro.io import AbstractDataSet, DataCatalog, MemoryDataSet, from kedro.pipeline import Pipeline and from kedro.runner.runner import AbstractRunner Signed-off-by: laizaparizotto <[email protected]> * Add logger object for logging when computing micropackage version Signed-off-by: laizaparizotto <[email protected]> * Added new format to the logging Signed-off-by: laizaparizotto <[email protected]> --------- Signed-off-by: laizaparizotto <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Try only trigger docs build when release notes updated (#2907) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add Python 3.11 support to Kedro (#2851) * add 3.11 Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * limit number of python version tests Signed-off-by: SajidAlamQB <[email protected]> * fix setup.py Signed-off-by: SajidAlamQB <[email protected]> * delta-spark version Signed-off-by: SajidAlamQB <[email protected]> * try original setup Signed-off-by: SajidAlamQB <[email protected]> * remove 3.11 from circleci Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * pin matplotlib to 3.7.2 for python 3.11 Signed-off-by: SajidAlamQB <[email protected]> * Update test_import.py Signed-off-by: SajidAlamQB <[email protected]> * add noks test adjustments Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * replace eq with is for multi_catalog Signed-off-by: SajidAlamQB <[email protected]> * minor fixes Signed-off-by: SajidAlamQB <[email protected]> * remove matplotlib 3.11 pin Signed-off-by: SajidAlamQB <[email protected]> * matplotlib pin for 3.11 Signed-off-by: SajidAlamQB <[email protected]> * run original matplotlib pin with ubuntu only Signed-off-by: SajidAlamQB <[email protected]> * run circleci Signed-off-by: SajidAlamQB <[email protected]> * pin spark < 3.4 as its breaking Signed-off-by: SajidAlamQB <[email protected]> * spark datasets revert earlier changes back to original Signed-off-by: SajidAlamQB <[email protected]> * delta Signed-off-by: SajidAlamQB <[email protected]> * remove all windows tests from GHA Signed-off-by: SajidAlamQB <[email protected]> * skip coverage as not needed Signed-off-by: SajidAlamQB <[email protected]> * Checking only WIN tests (will revert) Signed-off-by: SajidAlamQB <[email protected]> * Update unit-tests.yml Signed-off-by: SajidAlamQB <[email protected]> * undo previous commit Signed-off-by: SajidAlamQB <[email protected]> * add msbuild Signed-off-by: SajidAlamQB <[email protected]> * add ubuntu tests back in and remove 3.11 tests from circleci add in msbuild for the other tests Signed-off-by: SajidAlamQB <[email protected]> * set hdf5 disable check and lint Signed-off-by: SajidAlamQB <[email protected]> * pin tables for 3.8 Signed-off-by: SajidAlamQB <[email protected]> * pin 3.8 tables Signed-off-by: SajidAlamQB <[email protected]> * lint Signed-off-by: SajidAlamQB <[email protected]> * remove 3.11 completely from circleci and tidy up Signed-off-by: SajidAlamQB <[email protected]> * add 3.11 to pyarrow pin for docs Signed-off-by: SajidAlamQB <[email protected]> * undo previous commit Signed-off-by: SajidAlamQB <[email protected]> * update rtd to python 3.11 Signed-off-by: SajidAlamQB <[email protected]> * try 3.10 on rtd Signed-off-by: SajidAlamQB <[email protected]> * fix outdated pylint to ruff Signed-off-by: SajidAlamQB <[email protected]> * changes based on review Signed-off-by: SajidAlamQB <[email protected]> * add mock for pyarrow and deltalake for docs Signed-off-by: SajidAlamQB <[email protected]> * undo previous commit Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * Update .readthedocs.yml Signed-off-by: SajidAlamQB <[email protected]> * Update conf.py Signed-off-by: SajidAlamQB <[email protected]> * changes based on review Signed-off-by: SajidAlamQB <[email protected]> * fix comments Signed-off-by: SajidAlamQB <[email protected]> * test running 3.10 parallel Signed-off-by: SajidAlamQB <[email protected]> * Add to release notes and update badges on docs Signed-off-by: SajidAlamQB <[email protected]> * Add migration tips Signed-off-by: SajidAlamQB <[email protected]> --------- Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Revise FAQs and README (#2909) * Revise FAQs and README Signed-off-by: Jo Stichbury <[email protected]> * Add back the data layers FAQ as I've no idea where else it fits Signed-off-by: Jo Stichbury <[email protected]> * minor changes from review Signed-off-by: Jo Stichbury <[email protected]> --------- Signed-off-by: Jo Stichbury <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update Generator example (#2858) * update release note Signed-off-by: Nok <[email protected]> * update to binary mode when opening file Signed-off-by: Nok Chan <[email protected]> * Simplify the implementation of Dataset Signed-off-by: Nok Chan <[email protected]> * simplify the example - wip Signed-off-by: Nok Chan <[email protected]> * Update the nodes Signed-off-by: Nok Chan <[email protected]> * update example Signed-off-by: Nok Chan <[email protected]> * Update example Signed-off-by: Nok <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> * Apply suggestions from code review Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> * add more instruction to guide user to create the custom dataset Signed-off-by: Nok <[email protected]> * update the reference from `pipeline.py` -> `nodes.py` Signed-off-by: Nok <[email protected]> * Collapse the nodes.py function into a expandable section Signed-off-by: Nok <[email protected]> * Update nodes.md * Update nodes.md --------- Signed-off-by: Nok <[email protected]> Signed-off-by: Nok Chan <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * update docs to reflect change to /conf file structure (#2913) * update docs to reflect change to /conf file structure Signed-off-by: Dmitry Sorokin <[email protected]> * update docs, folder structure diagrams Signed-off-by: Dmitry Sorokin <[email protected]> --------- Signed-off-by: Dmitry Sorokin <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Change CONTRIBUTING.md file based on PR #2894 (#2914) Signed-off-by: lrcouto <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Move contribution guidelines from CONTRIBUTING.md to the Wiki (#2894) * Add drafts for new contributing.md and wiki page Signed-off-by: lrcouto <[email protected]> * Add contribution details to wiki page draft Signed-off-by: lrcouto <[email protected]> * Remove contributing-to-Kedro.md draft from the branch Signed-off-by: lrcouto <[email protected]> * Fix broken link to Prefect website in deploy guide (#2885) Signed-off-by: lrcouto <[email protected]> * Clarify the <micropkg_name> argument to kedro micropkg package (#2835) Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: lrcouto <[email protected]> * Allow registering of custom resolvers to `OmegaConfigLoader` (#2869) * Allow registering of custom resolvers to OCL Signed-off-by: Ankita Katiyar <[email protected]> * Complete doc string Signed-off-by: Ankita Katiyar <[email protected]> * Add test for overwritten resolvers Signed-off-by: Ankita Katiyar <[email protected]> * Update test for overwritten resolvers Signed-off-by: Ankita Katiyar <[email protected]> * Remove replace=True by default Signed-off-by: Ankita Katiyar <[email protected]> * Update release notes Signed-off-by: Ankita Katiyar <[email protected]> * Update release notes Signed-off-by: Ankita Katiyar <[email protected]> * Add debug level log for registering new resolver Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: lrcouto <[email protected]> * Some adjustments to ordering of text Signed-off-by: Jo Stichbury <[email protected]> Signed-off-by: lrcouto <[email protected]> * Document the use of custom resolvers with `OmegaConfigLoader` (#2896) * Update docs for custom resolvers for OCL Signed-off-by: Ankita Katiyar <[email protected]> * Update FAQ with correct link Signed-off-by: Ankita Katiyar <[email protected]> * Fix typo Co-authored-by: Merel Theisen <[email protected]> * Add note for oc.env Signed-off-by: Ankita Katiyar <[email protected]> * Suggestions from code review Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: lrcouto <[email protected]> * Update kedro pipeline create to use new /conf file structure (#2856) * add parameters_pipelineNane.yml template in the conf folder Signed-off-by: Dmitry Sorokin <[email protected]> * del yml template from parameters folder Signed-off-by: Dmitry Sorokin <[email protected]> * modify delete pipeline command to use correct path to yml files Signed-off-by: Dmitry Sorokin <[email protected]> * modify test_pipeline move parameters foldername to the yml filename Signed-off-by: Dmitry Sorokin <[email protected]> * collect configs to package not only from parameters folder, but from core conf folder also Signed-off-by: Dmitry Sorokin <[email protected]> * move parameters foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <[email protected]> * move parameters foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <[email protected]> * Add changes to RELEASE.md Signed-off-by: Dmitry Sorokin <[email protected]> * Update RELEASE.md Co-authored-by: Merel Theisen <[email protected]> * Update RELEASE.md, change to release 0.18.13 Signed-off-by: Dmitry Sorokin <[email protected]> * save old structure for 'pipeline delete' command for backward compatibility Signed-off-by: Dmitry Sorokin <[email protected]> * modify test_pipeline for 'pipeline delete' backward compability Signed-off-by: Dmitry Sorokin <[email protected]> --------- Signed-off-by: Dmitry Sorokin <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: lrcouto <[email protected]> * Update CONTRIBUTING.md Signed-off-by: lrcouto <[email protected]> * Fix broken link Signed-off-by: Jo Stichbury <[email protected]> Signed-off-by: lrcouto <[email protected]> * Update kedro catalog create to use new /conf structure (#2884) * modify test_catalog, remove nesting in the catalog folder Signed-off-by: Dmitry Sorokin <[email protected]> * move catalog foldername to the yml filename, remove nesting Signed-off-by: Dmitry Sorokin <[email protected]> * Update RELEASE.md Signed-off-by: Dmitry Sorokin <[email protected]> * Modify docs, fix broken link Signed-off-by: Dmitry Sorokin <[email protected]> --------- Signed-off-by: Dmitry Sorokin <[email protected]> Signed-off-by: lrcouto <[email protected]> * Add migration steps for `ConfigLoader` to `OmegaConfigLoader` (#2887) * Add migration steps for CL to OCL Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: lrcouto <[email protected]> * Fix #2498 Adding logging issue 2498 (#2842) * Fixing typo n on line 114 file /docs/source/nodes_and_pipelines/run_a_pipeline.md from issue #2607 Signed-off-by: laizaparizotto <[email protected]> * Removed redundant packages at lines 66, 67 and 68 on file kedro/docs/source/nodes_and_pipelines/run_a_pipeline.md, packages: from kedro.io import AbstractDataSet, DataCatalog, MemoryDataSet, from kedro.pipeline import Pipeline and from kedro.runner.runner import AbstractRunner Signed-off-by: laizaparizotto <[email protected]> * Add logger object for logging when computing micropackage version Signed-off-by: laizaparizotto <[email protected]> * Added new format to the logging Signed-off-by: laizaparizotto <[email protected]> --------- Signed-off-by: laizaparizotto <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: lrcouto <[email protected]> * Try only trigger docs build when release notes updated (#2907) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: lrcouto <[email protected]> * Add Python 3.11 support to Kedro (#2851) * add 3.11 Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * limit number of python version tests Signed-off-by: SajidAlamQB <[email protected]> * fix setup.py Signed-off-by: SajidAlamQB <[email protected]> * delta-spark version Signed-off-by: SajidAlamQB <[email protected]> * try original setup Signed-off-by: SajidAlamQB <[email protected]> * remove 3.11 from circleci Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * pin matplotlib to 3.7.2 for python 3.11 Signed-off-by: SajidAlamQB <[email protected]> * Update test_import.py Signed-off-by: SajidAlamQB <[email protected]> * add noks test adjustments Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * replace eq with is for multi_catalog Signed-off-by: SajidAlamQB <[email protected]> * minor fixes Signed-off-by: SajidAlamQB <[email protected]> * remove matplotlib 3.11 pin Signed-off-by: SajidAlamQB <[email protected]> * matplotlib pin for 3.11 Signed-off-by: SajidAlamQB <[email protected]> * run original matplotlib pin with ubuntu only Signed-off-by: SajidAlamQB <[email protected]> * run circleci Signed-off-by: SajidAlamQB <[email protected]> * pin spark < 3.4 as its breaking Signed-off-by: SajidAlamQB <[email protected]> * spark datasets revert earlier changes back to original Signed-off-by: SajidAlamQB <[email protected]> * delta Signed-off-by: SajidAlamQB <[email protected]> * remove all windows tests from GHA Signed-off-by: SajidAlamQB <[email protected]> * skip coverage as not needed Signed-off-by: SajidAlamQB <[email protected]> * Checking only WIN tests (will revert) Signed-off-by: SajidAlamQB <[email protected]> * Update unit-tests.yml Signed-off-by: SajidAlamQB <[email protected]> * undo previous commit Signed-off-by: SajidAlamQB <[email protected]> * add msbuild Signed-off-by: SajidAlamQB <[email protected]> * add ubuntu tests back in and remove 3.11 tests from circleci add in msbuild for the other tests Signed-off-by: SajidAlamQB <[email protected]> * set hdf5 disable check and lint Signed-off-by: SajidAlamQB <[email protected]> * pin tables for 3.8 Signed-off-by: SajidAlamQB <[email protected]> * pin 3.8 tables Signed-off-by: SajidAlamQB <[email protected]> * lint Signed-off-by: SajidAlamQB <[email protected]> * remove 3.11 completely from circleci and tidy up Signed-off-by: SajidAlamQB <[email protected]> * add 3.11 to pyarrow pin for docs Signed-off-by: SajidAlamQB <[email protected]> * undo previous commit Signed-off-by: SajidAlamQB <[email protected]> * update rtd to python 3.11 Signed-off-by: SajidAlamQB <[email protected]> * try 3.10 on rtd Signed-off-by: SajidAlamQB <[email protected]> * fix outdated pylint to ruff Signed-off-by: SajidAlamQB <[email protected]> * changes based on review Signed-off-by: SajidAlamQB <[email protected]> * add mock for pyarrow and deltalake for docs Signed-off-by: SajidAlamQB <[email protected]> * undo previous commit Signed-off-by: SajidAlamQB <[email protected]> * Update setup.py Signed-off-by: SajidAlamQB <[email protected]> * Update .readthedocs.yml Signed-off-by: SajidAlamQB <[email protected]> * Update conf.py Signed-off-by: SajidAlamQB <[email protected]> * changes based on review Signed-off-by: SajidAlamQB <[email protected]> * fix comments Signed-off-by: SajidAlamQB <[email protected]> * test running 3.10 parallel Signed-off-by: SajidAlamQB <[email protected]> * Add to release notes and update badges on docs Signed-off-by: SajidAlamQB <[email protected]> * Add migration tips Signed-off-by: SajidAlamQB <[email protected]> --------- Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: lrcouto <[email protected]> * Revise FAQs and README (#2909) * Revise FAQs and README Signed-off-by: Jo Stichbury <[email protected]> * Add back the data layers FAQ as I've no idea where else it fits Signed-off-by: Jo Stichbury <[email protected]> * minor changes from review Signed-off-by: Jo Stichbury <[email protected]> --------- Signed-off-by: Jo Stichbury <[email protected]> * Update Generator example (#2858) * update release note Signed-off-by: Nok <[email protected]> * update to binary mode when opening file Signed-off-by: Nok Chan <[email protected]> * Simplify the implementation of Dataset Signed-off-by: Nok Chan <[email protected]> * simplify the example - wip Signed-off-by: Nok Chan <[email protected]> * Update the nodes Signed-off-by: Nok Chan <[email protected]> * update example Signed-off-by: Nok Chan <[email protected]> * Update example Signed-off-by: Nok <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> * Apply suggestions from code review Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> * add more instruction to guide user to create the custom dataset Signed-off-by: Nok <[email protected]> * update the reference from `pipeline.py` -> `nodes.py` Signed-off-by: Nok <[email protected]> * Collapse the nodes.py function into a expandable section Signed-off-by: Nok <[email protected]> * Update nodes.md * Update nodes.md --------- Signed-off-by: Nok <[email protected]> Signed-off-by: Nok Chan <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: lrcouto <[email protected]> * Update CONTRIBUTING.md --------- Signed-off-by: lrcouto <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jo Stichbury <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: laizaparizotto <[email protected]> Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: Nok <[email protected]> Signed-off-by: Nok Chan <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Co-authored-by: Jonathan Cohen <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Ankita Katiyar <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Co-authored-by: Dmitry Sorokin <[email protected]> Co-authored-by: Laíza Milena Scheid Parizotto <[email protected]> Co-authored-by: Sajid Alam <[email protected]> Co-authored-by: Nok Lam Chan <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Remove redundant pages and direct users to wiki (#2917) * Remove redundant pages and direct users to wiki Signed-off-by: lrcouto <[email protected]> * Lint docs changes Signed-off-by: Jo Stichbury <[email protected]> * Few tweaks to index page Signed-off-by: Jo Stichbury <[email protected]> --------- Signed-off-by: lrcouto <[email protected]> Signed-off-by: Jo Stichbury <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Deprecate abstract "DataSet" in favor of "Dataset" (#2746) * Deprecate abstract "DataSet" in favor of "Dataset" Signed-off-by: Deepyaman Datta <[email protected]> * Update docs references to abstract "DataSet" class Signed-off-by: Deepyaman Datta <[email protected]> * Fix `AbstractDataSet` reference to `AbstractDataset` Signed-off-by: Deepyaman Datta <[email protected]> * Update docs/source/conf.py Signed-off-by: Deepyaman Datta <[email protected]> * Change remaining `_DEPRECATED_ERROR_CLASSES` to `_DEPRECATED_CLASSES`, update type hints Signed-off-by: Deepyaman Datta <[email protected]> * Update RELEASE.md * Reformat kedro/io/__init__.py with Black and isort Signed-off-by: Deepyaman Datta <[email protected]> * Remove duplicate imports Signed-off-by: Deepyaman Datta <[email protected]> * Move imports Signed-off-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Reorganise and improve the data catalog documentation (#2888) * First drop of newly organised data catalog docs Signed-off-by: Jo Stichbury <[email protected]> * linter Signed-off-by: Jo Stichbury <[email protected]> * Added to-do notes Signed-off-by: Jo Stichbury <[email protected]> * Afternoon's work in rewriting/reorganising content Signed-off-by: Jo Stichbury <[email protected]> * More changes Signed-off-by: Jo Stichbury <[email protected]> * Further changes Signed-off-by: Jo Stichbury <[email protected]> * Another chunk of changes Signed-off-by: Jo Stichbury <[email protected]> * Final changes Signed-off-by: Jo Stichbury <[email protected]> * Revise ordering of pages Signed-off-by: Jo Stichbury <[email protected]> * Add new CLI commands to dataset factory docs (#2935) * Add changes from #2930 Signed-off-by: Ahdra Merali <[email protected]> * Lint Signed-off-by: Ahdra Merali <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> * Make code snippets collapsable Signed-off-by: Ahdra Merali <[email protected]> --------- Signed-off-by: Ahdra Merali <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> * Bunch of changes from feedback Signed-off-by: Jo Stichbury <[email protected]> * A few more tweaks Signed-off-by: Jo Stichbury <[email protected]> * Update h1,h2,h3 font sizes Signed-off-by: Tynan DeBold <[email protected]> * Add code snippet for using DataCatalog with Kedro config Signed-off-by: Ankita Katiyar <[email protected]> * Few more tweaks Signed-off-by: Jo Stichbury <[email protected]> * Update docs/source/data/data_catalog.md * Upgrade kedro-datasets for docs Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> * Improve prose Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> --------- Signed-off-by: Jo Stichbury <[email protected]> Signed-off-by: Ahdra Merali <[email protected]> Signed-off-by: Tynan DeBold <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> Co-authored-by: Ahdra Merali <[email protected]> Co-authored-by: Tynan DeBold <[email protected]> Co-authored-by: Ankita Katiyar <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add line about Viz to PR template (#2929) Signed-off-by: Jason Hite <[email protected]> * Add architecture graphic back to docs with revisions (#2916) * Revise FAQs and README Signed-off-by: Jo Stichbury <[email protected]> * Add back the data layers FAQ as I've no idea where else it fits Signed-off-by: Jo Stichbury <[email protected]> * minor changes from review Signed-off-by: Jo Stichbury <[email protected]> * Re-add kedro arch diagram, with revised graphic Signed-off-by: Jo Stichbury <[email protected]> * fix broken anchor Signed-off-by: Jo Stichbury <[email protected]> * fix broken anchor Signed-off-by: Jo Stichbury <[email protected]> * Update docs/source/extend_kedro/architecture_overview.md Co-authored-by: Nok Lam Chan <[email protected]> * Update docs/source/extend_kedro/architecture_overview.md Co-authored-by: Nok Lam Chan <[email protected]> * Update docs/source/extend_kedro/architecture_overview.md Co-authored-by: Nok Lam Chan <[email protected]> * Update docs/source/extend_kedro/architecture_overview.md Co-authored-by: Nok Lam Chan <[email protected]> * Changes to architecture page following review Signed-off-by: Jo Stichbury <[email protected]> * Change diagram following reivew * Add links to API docs Signed-off-by: Jo Stichbury <[email protected]> * Added in type of users * Fix linting error Signed-off-by: Jo Stichbury <[email protected]> --------- Signed-off-by: Jo Stichbury <[email protected]> Co-authored-by: Nok Lam Chan <[email protected]> Co-authored-by: Yetunde Dada <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add kedro catalog resolve command (#2891) * Implement working draft Signed-off-by: Ahdra Merali <[email protected]> * Remove comments Signed-off-by: Ahdra Merali <[email protected]> * Remove raw_config Signed-off-by: Ahdra Merali <[email protected]> * Use config loader to access catalog config Signed-off-by: Ahdra Merali <[email protected]> * Use dict comprehension Signed-off-by: Ahdra Merali <[email protected]> * Remove pipeline filtering Signed-off-by: Ahdra Merali <[email protected]> * Prevent overwrite or param inclusion Signed-off-by: Ahdra Merali <[email protected]> * Trim filepath Signed-off-by: Ahdra Merali <[email protected]> * Appease linter Signed-off-by: Ahdra Merali <[email protected]> * Add test for resolve Signed-off-by: Ahdra Merali <[email protected]> * Add test to exclude params Signed-off-by: Ahdra Merali <[email protected]> * Add test for overwrite (not working) Signed-off-by: Ahdra Merali <[email protected]> * Fix test Signed-off-by: Ahdra Merali <[email protected]> * Remove print Signed-off-by: Ahdra Merali <[email protected]> * Add changes to RELEASE.md Signed-off-by: Ahdra Merali <[email protected]> * Trim unreachable code Signed-off-by: Ahdra Merali <[email protected]> * Make helper function private Signed-off-by: Ahdra Merali <[email protected]> * Refactor Signed-off-by: Ahdra Merali <[email protected]> * Add changes to the docs Signed-off-by: Ahdra Merali <[email protected]> * Refactor 2 Signed-off-by: Ahdra Merali <[email protected]> * Fix docs build Signed-off-by: Ahdra Merali <[email protected]> * Add suggestions from code review (docs) Signed-off-by: Ahdra Merali <[email protected]> * Add suggestions from code review (code) Signed-off-by: Ahdra Merali <[email protected]> * Rename catalog_config variable to explicit_datasets Signed-off-by: Ahdra Merali <[email protected]> * Apply suggestions from code review Co-authored-by: Merel Theisen <[email protected]> * Fix mocking in test Signed-off-by: Ahdra Merali <[email protected]> * Change test fixture name Signed-off-by: Ahdra Merali <[email protected]> * Lint Signed-off-by: Ahdra Merali <[email protected]> --------- Signed-off-by: Ahdra Merali <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Replace "DataSet" with "Dataset" in Markdown files (#2735) * LambdaDataSet->LambdaDataset in .md files Signed-off-by: Deepyaman Datta <[email protected]> * MemoryDataSet->MemoryDataset in .md files Signed-off-by: Deepyaman Datta <[email protected]> * PartitionedDataSet->PartitionedDataset in .md files Signed-off-by: Deepyaman Datta <[email protected]> * IncrementalDataSet->IncrementalDataset in .md files Signed-off-by: Deepyaman Datta <[email protected]> * CachedDataSet->CachedDataset in .md files Signed-off-by: Deepyaman Datta <[email protected]> * DataSetError->DatasetError in .md files Signed-off-by: Deepyaman Datta <[email protected]> * DataSetNotFoundError->DatasetNotFoundError in .md files Signed-off-by: Deepyaman Datta <[email protected]> * Replace "DataSet" with "Dataset" in Markdown files * Update RELEASE.md * Fix remaining instance of "*DataSet*"->"*Dataset*" Signed-off-by: Deepyaman Datta <[email protected]> * `find . -name '*.md' -print0 | xargs -0 sed -i "" "s/\([^A-Za-z]\)DataSet/\1Dataset/g"` Signed-off-by: Deepyaman Datta <[email protected]> * Change non-class instances of Dataset to dataset * Replace any remaining instances of DataSet in docs * Fix a broken link to docs for `PartitionedDataset` --------- Signed-off-by: Deepyaman Datta <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update h1,h2,h3 font sizes in the docs pages (#2938) * Update h1,h2,h3 font sizes Signed-off-by: Tynan DeBold <[email protected]> * Update h2 font size Signed-off-by: Tynan DeBold <[email protected]> --------- Signed-off-by: Tynan DeBold <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Automatically trigger `kedro-starters` release on the release of `kedro` (#2889) * Add GA for triggering starters release Signed-off-by: Ankita Katiyar <[email protected]> * Update token name Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Create issues_metrics.yml (#2814) * Create issues_metrics.yml * Update issues_metrics.yml Signed-off-by: Jason Hite <[email protected]> * Clean up setuptools and wheel requirements to align with PEP-518 (#2927) Signed-off-by: Jason Hite <[email protected]> * Clean up `kedro pipeline create` outdated docs (#2945) * remove the README template generate by kedro pipeline create Signed-off-by: Nok <[email protected]> * fix docstring for kedro catalog create Signed-off-by: Nok <[email protected]> * Fix broken link Signed-off-by: Nok <[email protected]> * fix broken link Signed-off-by: Nok <[email protected]> --------- Signed-off-by: Nok <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add globals feature for `OmegaConfigLoader` using a globals resolver (#2921) * Refactor load_and_merge_dir() Signed-off-by: Ankita Katiyar <[email protected]> * Try adding globals resolver Signed-off-by: Ankita Katiyar <[email protected]> * Minor change Signed-off-by: Ankita Katiyar <[email protected]> * Add globals resolver Signed-off-by: Ankita Katiyar <[email protected]> * Revert refactoring Signed-off-by: Ankita Katiyar <[email protected]> * Add test + remove self.globals Signed-off-by: Ankita Katiyar <[email protected]> * Allow for nested variables in globals Signed-off-by: Ankita Katiyar <[email protected]> * Add documentation Signed-off-by: Ankita Katiyar <[email protected]> * Typo Signed-off-by: Ankita Katiyar <[email protected]> * Add error message + test Signed-off-by: Ankita Katiyar <[email protected]> * Apply suggestions from code review Co-authored-by: Merel Theisen <[email protected]> * Split test into multiple tests Signed-off-by: Ankita Katiyar <[email protected]> * Restrict the globals config_patterns Signed-off-by: Ankita Katiyar <[email protected]> * Release notes Signed-off-by: Ankita Katiyar <[email protected]> * Update docs/source/configuration/advanced_configuration.md Co-authored-by: Merel Theisen <[email protected]> * Add helpful error message for keys starting with _ Signed-off-by: Ankita Katiyar <[email protected]> * Enable setting default value for globals resolver Signed-off-by: Ankita Katiyar <[email protected]> * Typo Signed-off-by: Ankita Katiyar <[email protected]> * Move test for keys starting with _ to the top Signed-off-by: Ankita Katiyar <[email protected]> * Fix cross ref link in docs Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Co-authored-by: Merel Theisen <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Co-authored-by: Nok Lam Chan <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Consolidate two `ruff-pre-commit` entries into one (#2881) * Clean up Ruff configuration in base pyproject.toml Signed-off-by: Deepyaman Datta <[email protected]> * Consolidate two `ruff-pre-commit` entries into one Signed-off-by: Deepyaman Datta <[email protected]> * Move Ruff hook to the top, because it's aggressive Signed-off-by: Deepyaman Datta <[email protected]> * Add `kedro` as `known-first-party` for `isort` run Signed-off-by: Deepyaman Datta <[email protected]> * Add noqa in docs/source/conf.py Signed-off-by: Deepyaman Datta <[email protected]> * Fix ruff linter precommit config Signed-off-by: Nok <[email protected]> --------- Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Nok <[email protected]> Co-authored-by: Nok <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Fix typos across the documentation (#2956) * Fix typos across docs Signed-off-by: Ankita Katiyar <[email protected]> * Capitalisation stuff Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Setup Vale linter as a GHA workflow (#2953) * Add vale GHA Signed-off-by: Ankita Katiyar <[email protected]> * Update condition Signed-off-by: Ankita Katiyar <[email protected]> * Update .github/styles/Kedro/ignore.txt Co-authored-by: Jo Stichbury <[email protected]> * add some styles Signed-off-by: Jo Stichbury <[email protected]> * Remove styles for now Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jo Stichbury <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Fix README to show graphics on PyPI (#2961) * Fix for PyPI Signed-off-by: Jo Stichbury <[email protected]> * Few more tweaks Signed-off-by: Jo Stichbury <[email protected]> * Few more tweaks Signed-off-by: Jo Stichbury <[email protected]> * Few more tweaks Signed-off-by: Jo Stichbury <[email protected]> * Few more tweaks Signed-off-by: Jo Stichbury <[email protected]> * Update README.md * Update README.md --------- Signed-off-by: Jo Stichbury <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add some Vale styles (#2963) * add some styles Signed-off-by: Jo Stichbury <[email protected]> * Tweaks to styles Signed-off-by: Jo Stichbury <[email protected]> * more tweaks Signed-off-by: Jo Stichbury <[email protected]> * Revise some styles Signed-off-by: Jo Stichbury <[email protected]> * Revise severity of some styles Signed-off-by: Jo Stichbury <[email protected]> * Update .github/styles/Kedro/words.yml Co-authored-by: Deepyaman Datta <[email protected]> --------- Signed-off-by: Jo Stichbury <[email protected]> Co-authored-by: Deepyaman Datta <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Minor changes to test + release notes + docs Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update anyconfig requirement from ~=0.10.0 to >=0.10,<0.14 (#2876) Updates the requirements on [anyconfig](https://github.com/ssato/python-anyconfig) to permit the latest version. - [Changelog](https://github.com/ssato/python-anyconfig/blob/next/NEWS) - [Commits](https://github.com/ssato/python-anyconfig/compare/RELEASE_0.10.0...RELEASE_0.13.0) Signed-off-by: Jason Hite <[email protected]> --- updated-dependencies: - dependency-name: anyconfig dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Move default template to static `pyproject.toml`, take 2 (#2853) * Remove leftover reference to micropkg setup.py Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> * Replace outdated extras requirement with kedro-datasets Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> * Remove outdated mention to environment.yml Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> * Move flake8 configuration to dedicated file Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> * Replace setup.py with pyproject.toml in default project template Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> * Adapt link to moving file in docs Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> * Revert "Replace outdated extras requirement with kedro-datasets" This reverts commit dfa8a9e8377ca60dcc3026e93087c5538791a0b5. Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> * Add release notes for pyproject.toml change Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> --------- Signed-off-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add deprecation warnings to top-level use of layer in catalog definition (#2964) * add deprecation Signed-off-by: SajidAlamQB <[email protected]> * lint Signed-off-by: SajidAlamQB <[email protected]> * Use FutureWarning Signed-off-by: SajidAlamQB <[email protected]> * changes based on review Signed-off-by: SajidAlamQB <[email protected]> * add link to docs Signed-off-by: SajidAlamQB <[email protected]> * Update RELEASE.md Co-authored-by: Jo Stichbury <[email protected]> --------- Signed-off-by: SajidAlamQB <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update on credentials.md (#2787) * Update on credentials.md Updating example code in credentials.md to make it usable when copy/pasting. Since as of now it will generate a TypeError Signed-off-by: Jose <[email protected]> * adding a more explicit code example, now we directly show the user that project_path is a pathlib.Path object Signed-off-by: Jose <[email protected]> * Update docs/source/configuration/credentials.md Co-authored-by: Jo Stichbury <[email protected]> --------- Signed-off-by: Jose <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Co-authored-by: Nok Lam Chan <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Cap pluggy 1.3 release (#2981) Signed-off-by: Dmitry Sorokin <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Make vale linter only run when PR opened or reopened (#2982) Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update merge-gatekeeper.yml (#2960) Signed-off-by: Jason Hite <[email protected]> * Configure starters to use OmegaConfigLoader (#2974) * Change starters to use OmegaConfigLoader Signed-off-by: lrcouto <[email protected]> * Fix linter error Signed-off-by: lrcouto <[email protected]> * Suppress import outside toplevel linting for starters template Signed-off-by: L. R. Couto <[email protected]> * Fix linter error Signed-off-by: lrcouto <[email protected]> * Add changes to release notes Signed-off-by: lrcouto <[email protected]> --------- Signed-off-by: lrcouto <[email protected]> Signed-off-by: L. R. Couto <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Stop OmegaConfigLoader from reading config from hidden directory like `ipynb_checkpoints` (#2977) * Check plugins implement valid hooks Signed-off-by: Nok Chan <[email protected]> * Add release note Signed-off-by: Nok Chan <[email protected]> * Staging work - add custom functions to check hidden folder and files. Tests still failing Signed-off-by: Nok Chan <[email protected]> * Fix test - checkpoints should use the same environment Signed-off-by: Nok Chan <[email protected]> * Revert "Check plugins implement valid hooks" This reverts commit f10bede0401b80e3385e93458995835a2db721c9. * Update RELEASE.md Co-authored-by: Ankita Katiyar <[email protected]> * fix lint Signed-off-by: Nok <[email protected]> --------- Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok <[email protected]> Co-authored-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add migration steps for `TemplatedConfigLoader` to `OmegaConfigLoader` (#2904) Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Introduce a sentinel value _NO_VALUE to improve Global resolvers to support defaults `0` or `None` (#2976) * Add None support to globals Signed-off-by: Ankita Katiyar <[email protected]> * Add warning when default value is used Signed-off-by: Ankita Katiyar <[email protected]> * Check keys Signed-off-by: Ankita Katiyar <[email protected]> * Nok's suggestions Signed-off-by: Ankita Katiyar <[email protected]> * Create the test to check the non-existing keys Signed-off-by: Nok <[email protected]> * add more tests to catch case when global key is not a dict Signed-off-by: Nok <[email protected]> * Fix the null test Signed-off-by: Nok <[email protected]> * Introduce sentinel value _NO_VALUE Signed-off-by: Nok <[email protected]> * rename test Signed-off-by: Nok <[email protected]> * Improve error mesasge and raise InterpolationResolutionError when key does not exist and no default Signed-off-by: Nok <[email protected]> * Fix non exist default test Signed-off-by: Nok <[email protected]> * Fix test Signed-off-by: Nok <[email protected]> * Use omegaconf to replace the custom resolving logic Signed-off-by: Nok <[email protected]> * uncommented some tests Signed-off-by: Nok <[email protected]> * Remove dead code Signed-off-by: Ankita Katiyar <[email protected]> * Update error message Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Nok <[email protected]> Co-authored-by: Nok <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Release 0.18.13 (#2988) Signed-off-by: Jason Hite <[email protected]> * Fix docstrings on kedro/extras/datasets (#2995) * Fix docstrings on kedro/extras/datasets Signed-off-by: lrcouto <[email protected]> * Fix formatting error Signed-off-by: lrcouto <[email protected]> * Explicitly define code block Signed-off-by: lrcouto <[email protected]> * Fix empty line under code block Signed-off-by: lrcouto <[email protected]> * Fix formatting error Signed-off-by: lrcouto <[email protected]> * Fix broken link Signed-off-by: lrcouto <[email protected]> * Bump kedro-datasets version Signed-off-by: Ankita Katiyar <[email protected]> * Update links in partitioned and incremental datasets Signed-off-by: Ankita Katiyar <[email protected]> * Update links in partitioned and incremental datasets Signed-off-by: Ankita Katiyar <[email protected]> * Update links in partitioned dataset Signed-off-by: Ankita Katiyar <[email protected]> * Update links in partitioned dataset Signed-off-by: Ankita Katiyar <[email protected]> * Add polars.GenericDataSet to .rst Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: lrcouto <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]> Co-authored-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Minor docs changes on data section to create a PR and test Vale styles (#2966) * Minor changes to create a PR and test Vale styles Signed-off-by: Jo Stichbury <[email protected]> * fix some vale warnings Signed-off-by: Jo Stichbury <[email protected]> --------- Signed-off-by: Jo Stichbury <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * PiP/pyproject.toml and Conda/Meta.yaml Sync (#2922) * Resynced Signed-off-by: rxm7706 <[email protected]> * restore lower bound for pluggy Signed-off-by: rxm7706 <[email protected]> --------- Signed-off-by: rxm7706 <[email protected]> Co-authored-by: Juan Luis Cano Rodríguez <[email protected]> Signed-off-by: Jason Hite <[email protected]> * fix typo for build (#3001) Signed-off-by: rxm7706 <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Add hook example to access `metadata` (#2998) * Check plugins implement valid hooks Signed-off-by: Nok Chan <[email protected]> * add Metadtahook Signed-off-by: Nok Chan <[email protected]> * Fix docs according to comments Signed-off-by: Nok <[email protected]> --------- Signed-off-by: Nok Chan <[email protected]> Signed-off-by: Nok <[email protected]> Co-authored-by: Jo Stichbury <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Bump release notes version Signed-off-by: Jason Hite <[email protected]> * Expand docs for customized pipeline templates. Signed-off-by: Jason Hite <[email protected]> * Update release notes and docs Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update language linter to also run when PR converted ready for review Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Fix ci - Use `--resolver=backtracking` with `kedro build-reqs` in e2e tests (#3010) * Bump build version Signed-off-by: Ankita Katiyar <[email protected]> * Remove telemetry from test default starter Signed-off-by: Ankita Katiyar <[email protected]> * Add package_name back Signed-off-by: Ankita Katiyar <[email protected]> * Pin build only for 3.7 Signed-off-by: Ankita Katiyar <[email protected]> * Try upgrade pip Signed-off-by: Ankita Katiyar <[email protected]> * Add constraint Signed-off-by: Ankita Katiyar <[email protected]> * Update for windows Signed-off-by: Ankita Katiyar <[email protected]> * Run build-reqs with backtracking resolver Signed-off-by: Ankita Katiyar <[email protected]> * Run build-reqs with backtracking resolver Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update docs Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Update style to catch some more US spellings Signed-off-by: Jo Stichbury <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Make dataset factory resolve nested dict properly (#2993) Signed-off-by: Jason Hite <[email protected]> * Apply suggestions from code review Co-authored-by: Jo Stichbury <[email protected]> Signed-off-by: Jason Hite <[email protected]> * Resolve suggestions from code review Signed-off-by: Jason Hite <[email protected]> * Revert changes ukspelling.yml Signed-off-by: Ankita Katiyar <[email protected]> --------- Signed-off-by: Jason Hite <[email protected]> Signed-off-by: SajidAlamQB <[email protected]> Signed-off-by: Ankita Katiyar <[email protected]> Signed-off-by: Dmitry Sorokin <[email protected]> Signed-off-by: Merel Theisen <[email protected]> Signed-off-by: laizaparizotto <[email protected]> Signed-off-by: Jo Stichbury <[email protected]> Signed-off-by: Nok <[email protected]> Signed-off-by: Nok Chan <[email protected]> Signed-off-by: lrcouto <[email protected]> Signed-off-by: Deepyaman Datta <[email protected]> Signed-off-by: Ahdra Merali <ahdra.merali…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Second part of #2699
Development notes
Left to do:
OmegaConfigLoaderusing custom "globals" resolver #27940.18.Xto version that includes globals forOmegaConfigLoader, probably0.18.13.OmegaConfigLoaderusing custom "globals" resolver #2794 is mergedQuestions for reviewers:
ConfigLoadertoOmegaConfigLoaderandTemplatedConfigLoadertoOmegaConfigLoader. I just repeated them, because I think users will just skip the first section if they're only interested inTemplatedConfigLoader. Does that seem right?Checklist
RELEASE.mdfile