Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate dataset errors with module __getattr__ #2673

Merged
merged 108 commits into from
Jun 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
bcf66a5
Replace and deprecate `DataSet` use in class names
deepyaman Apr 10, 2023
87172c0
Replace another format string with an f-string
deepyaman Apr 10, 2023
af5f4a1
Perform deprecations for cached, lambda, and partitioned datasets
deepyaman Apr 10, 2023
53eaf4c
Deprecated `MemoryDataSet` in favor of `MemoryDataset`
deepyaman Apr 10, 2023
6377e12
Fix keyword argument to specify metaclass on `CachedDataSet`
deepyaman Apr 10, 2023
82b99f1
Fix reference to `PartitionedDataset`
deepyaman Apr 10, 2023
a8a0da6
Keep `AbstractDataSet` subscriptable
deepyaman Apr 10, 2023
0f1b3be
Update __init__.py files, __all__ definitions, etc
deepyaman Apr 27, 2023
ccdf78e
Merge branch 'main' into rename-data-set
deepyaman Apr 27, 2023
e738c32
Warn of impending Kedro 0.19 (not abstract future)
deepyaman Apr 27, 2023
7f02f86
Update `VideoDataSet` to `VideoDataset` (and refs)
deepyaman Apr 27, 2023
64b1a76
Merge branch 'rename-data-set' of https://github.com/kedro-org/kedro …
deepyaman Apr 27, 2023
cae65ca
Add missing `kedro.utils.DeprecatedClassMeta` imps
deepyaman Apr 27, 2023
904b5d6
Change deprecated references to `AbstractDataSet`
deepyaman Apr 27, 2023
f5654a7
Warn of impending Kedro 0.19 (not abstract future)
deepyaman Apr 28, 2023
139c4b0
Rename `pandas.CSVDataSet` to `pandas.CSVDataset`
deepyaman Apr 28, 2023
686db58
Fix some pylint errors and blacken code
deepyaman Apr 28, 2023
f3359fc
Update `dask.ParquetDataSet`
deepyaman Apr 28, 2023
2987744
Undo changes for `VideoDataSet`, inherit from new base
deepyaman Apr 28, 2023
5d92194
Undo changes to `APIDataSet`, inherit from new base
deepyaman Apr 28, 2023
4fde3d7
Fix some imports and missed references
deepyaman Apr 28, 2023
c6c2177
Undo changes to `BioSequenceDataSet`, inherit from new base
deepyaman Apr 28, 2023
1ef2ee2
Undo changes to Dask and Pandas datasets, inherit from new bases
deepyaman Apr 28, 2023
50e7f5e
Merge branch 'main' into rename-data-set
deepyaman Apr 28, 2023
7fff5a7
Remove the `AbstractDataset` and `AbstractVersionedDataset` alias, up…
deepyaman Apr 28, 2023
a2c5ae9
Merge branch 'rename-data-set' of https://github.com/kedro-org/kedro …
deepyaman Apr 28, 2023
c332aee
Undo changes in `kedro/extras/datasets`
deepyaman Apr 28, 2023
f371bfb
Update branch
deepyaman Apr 28, 2023
bdc5e1d
Change `DataSetError` to `DatasetError`
deepyaman Apr 28, 2023
cfd3e98
Remove deprecated aliases for Abstract*DataSet
deepyaman Apr 28, 2023
c545af6
Change `DataSetError` to `DatasetError` in tests/
deepyaman Apr 30, 2023
54c4def
Change DataSet*Error to Dataset*Error in tests/
deepyaman Apr 30, 2023
c14d4cc
Fix references to DataSet in a lot of tests
deepyaman May 1, 2023
7fafedf
Change `CSVDataset` back to `CSBVDataSet`
deepyaman May 2, 2023
6c5938c
Merge branch 'main' into rename-data-set
deepyaman May 20, 2023
fe8381e
Merge branch 'main' into rename-data-set
deepyaman Jun 1, 2023
edf5424
Rename core datasets used across `tests` directory
deepyaman Jun 1, 2023
8fec4da
Fix "Saving 'None' to a 'DataSet' is not allowed." messages
deepyaman Jun 1, 2023
81cb042
Fix `test_http_filesystem_no_versioning` everywhere
deepyaman Jun 1, 2023
524dd62
Fix removal of "data"
deepyaman Jun 1, 2023
35a6614
Deprecate `_SharedMemoryDataSet` in favor of `_SharedMemoryDataset`
deepyaman Jun 1, 2023
0fb89f0
Fix tests/pipeline/test_pipeline_from_missing.py
deepyaman Jun 1, 2023
0732f66
Fix list datasets test
deepyaman Jun 1, 2023
2470009
Change patched IncrementalDataSet to IncrementalDataset
deepyaman Jun 1, 2023
3817ee9
Fix default checkpoint dataset
deepyaman Jun 1, 2023
9ff515e
Fix data catalog tests
deepyaman Jun 1, 2023
a98fcff
Fix error message
deepyaman Jun 1, 2023
612f891
Merge branch 'main' into rename-data-set
deepyaman Jun 1, 2023
e1d4687
Use `MemoryDataset`, not `MemoryDataSet`, by default
deepyaman Jun 1, 2023
e57f2ac
Use `MemoryDataset`, not `MemoryDataSet`, for missing datasets in dat…
deepyaman Jun 1, 2023
d70e91b
Rename DefaultDataSet key to DefaultDataset
deepyaman Jun 1, 2023
13474bd
Change `LambdaDataSet` to `LambdaDataset` in `test_node_run.py`
deepyaman Jun 1, 2023
b408627
Update error message--but should I?
deepyaman Jun 1, 2023
c23a85b
Update error message--but should I?
deepyaman Jun 1, 2023
5f5a4b3
Update error message in kedro/io/core.py--but should I?
deepyaman Jun 1, 2023
7c01baa
Update RELEASE.md
deepyaman Jun 1, 2023
833fadd
Fix remaining tests
deepyaman Jun 2, 2023
796dadd
Fix lint issues
deepyaman Jun 2, 2023
aa877f2
Align capitalization
deepyaman Jun 2, 2023
452ad0b
Add `DeprecatedClassMeta` tests from StackOverflow
deepyaman Jun 5, 2023
0fd5a2e
Blacken kedro/utils.py
deepyaman Jun 5, 2023
1053cd8
Ignore "No value for argument 'subclass' in unbound method call"
deepyaman Jun 5, 2023
1b7a1bc
Rename 'foo' to 'value' to satisfy linter
deepyaman Jun 5, 2023
41436a8
Disable pylint messages on deprecated class definitions
deepyaman Jun 5, 2023
b8de711
Merge branch 'main' into rename-data-set
deepyaman Jun 5, 2023
f19b575
Blacken kedro/utils.py
deepyaman Jun 5, 2023
db9b7ac
Wrap `kedro.io.core` to fix error deprecation
deepyaman Jun 5, 2023
e9294be
Simplify deprecation of error names to try to fix docs
deepyaman Jun 5, 2023
1ca493a
Undo attempt to make docs pass
deepyaman Jun 5, 2023
7b65cbd
Merge branch 'main' into rename-data-set
deepyaman Jun 5, 2023
68a08ce
Merge branch 'main' into rename-data-set
deepyaman Jun 6, 2023
5ef91bb
Merge branch 'main' into rename-data-set
deepyaman Jun 7, 2023
5ad59dd
Merge branch 'main' into rename-data-set
deepyaman Jun 9, 2023
2d41aec
Merge branch 'main' into rename-data-set
deepyaman Jun 12, 2023
9bcac42
Wrap `kedro.io.core` to fix error deprecation
deepyaman Jun 12, 2023
a9fbde1
Leverage PEP 562 for less hacky error deprecations
deepyaman Jun 12, 2023
71da589
Test old `DataSetError` in `kedro.extras.datasets`
deepyaman Jun 12, 2023
9c79843
Fix missing and unnecessary imports/other mistakes
deepyaman Jun 12, 2023
a84cbf0
Blacken kedro/io/core.py
deepyaman Jun 12, 2023
c0ebba2
Don't raise `DeprecationWarning` each time import from `kedro.io`
deepyaman Jun 12, 2023
582bf9a
Merge branch 'main' into rename-data-set
stichbury Jun 13, 2023
71b7a6a
Merge branch 'main' into rename-data-set
deepyaman Jun 15, 2023
3af6a03
Replace `DataSetError` with `DatasetError` in test
deepyaman Jun 15, 2023
355c41c
Add missing "in" to a `DeprecationWarning` message
deepyaman Jun 15, 2023
5b17331
Add "Dataset" versions of errors to `kedro.io` doc
deepyaman Jun 15, 2023
2120812
Add updated "Dataset" names to `kedro.io.rst` and sort the entries
deepyaman Jun 15, 2023
32b2bb5
Add `_SharedMemoryDataset` to type targets in conf
deepyaman Jun 15, 2023
5f82b75
Merge branch 'rename-data-set' into refactor/rename-data-set
deepyaman Jun 15, 2023
e692d88
Revert all changes to `DatasetError` in `kedro/extras/datasets`
deepyaman Jun 15, 2023
f01db94
docs(datasets): fix a ref to `GeoJSONLocalDataset`
deepyaman Jun 15, 2023
22575f7
Consistently use `DatasetError` in dataset tests
deepyaman Jun 15, 2023
7b3b115
Merge branch 'main' into refactor/rename-data-set
deepyaman Jun 16, 2023
b9baf18
Merge branch 'refactor/rename-data-set' of https://github.com/kedro-o…
deepyaman Jun 16, 2023
cb43180
Remove unused import
deepyaman Jun 16, 2023
5f7d8bf
Remove "DataSet" imports from kedro/io/__init__.py
deepyaman Jun 16, 2023
6699e45
chore(sdk): use type hints to skirt linting errors
deepyaman Jun 17, 2023
9582a99
Merge branch 'main' into refactor/rename-data-set
deepyaman Jun 17, 2023
93dfaa9
Implement workaround in `kedro/io/__init__.py` too
deepyaman Jun 17, 2023
6cb7096
Merge branch 'refactor/rename-data-set' of https://github.com/kedro-o…
deepyaman Jun 17, 2023
0ab4fae
Import future annotation in `kedro/io/__init__.py`
deepyaman Jun 17, 2023
36ef08c
test(datasets): cover raising a DeprecationWarning
deepyaman Jun 18, 2023
30c4cdd
Ignore pylint warnings (working as intended)
deepyaman Jun 20, 2023
634bb09
Merge branch 'main' into refactor/rename-data-set
deepyaman Jun 20, 2023
9b13345
Replace `globals()` in `kedro.io.core.__getattr__`
deepyaman Jun 20, 2023
f5d3b70
Remove use of `exec` (use `import_module` instead)
deepyaman Jun 20, 2023
f316afe
Rename the parameters passed to `test_deprecation`
deepyaman Jun 20, 2023
dab939e
Merge branch 'main' into refactor/rename-data-set
tynandebold Jun 21, 2023
8fd92f0
Merge branch 'main' into refactor/rename-data-set
deepyaman Jun 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion kedro/extras/datasets/geopandas/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""``GeoJSONLocalDataset`` is an ``AbstractVersionedDataSet`` to save and load GeoJSON files.
"""``GeoJSONDataSet`` is an ``AbstractVersionedDataSet`` to save and load GeoJSON files.
"""
__all__ = ["GeoJSONDataSet"]

Expand Down
20 changes: 17 additions & 3 deletions kedro/io/__init__.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
"""``kedro.io`` provides functionality to read and write to a
number of data sets. At the core of the library is the ``AbstractDataSet`` class.
"""
from __future__ import annotations

from .cached_dataset import CachedDataSet, CachedDataset
from .core import (
AbstractDataSet,
AbstractVersionedDataSet,
DataSetAlreadyExistsError,
DatasetAlreadyExistsError,
DataSetError,
DatasetError,
DataSetNotFoundError,
DatasetNotFoundError,
Version,
)
Expand All @@ -24,6 +22,22 @@
PartitionedDataset,
)

# https://github.com/pylint-dev/pylint/issues/4300#issuecomment-1043601901
DataSetError: type[Exception]
DataSetNotFoundError: type[DatasetError]
DataSetAlreadyExistsError: type[DatasetError]


def __getattr__(name):
import kedro.io.core # pylint: disable=import-outside-toplevel

if name in (
kedro.io.core._DEPRECATED_ERROR_CLASSES # pylint: disable=protected-access
):
return getattr(kedro.io.core, name)
astrojuanlu marked this conversation as resolved.
Show resolved Hide resolved
raise AttributeError(f"module {repr(__name__)} has no attribute {repr(name)}")
astrojuanlu marked this conversation as resolved.
Show resolved Hide resolved


__all__ = [
"AbstractDataSet",
"AbstractVersionedDataSet",
Expand Down
39 changes: 23 additions & 16 deletions kedro/io/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from cachetools import Cache, cachedmethod
from cachetools.keys import hashkey

from kedro.utils import DeprecatedClassMeta, load_obj
from kedro.utils import load_obj

warnings.simplefilter("default", DeprecationWarning)

Expand All @@ -31,6 +31,11 @@
PROTOCOL_DELIMITER = "://"
CLOUD_PROTOCOLS = ("s3", "s3n", "s3a", "gcs", "gs", "adl", "abfs", "abfss", "gdrive")

# https://github.com/pylint-dev/pylint/issues/4300#issuecomment-1043601901
DataSetError: type[Exception]
DataSetNotFoundError: type[DatasetError]
DataSetAlreadyExistsError: type[DatasetError]


class DatasetError(Exception):
"""``DatasetError`` raised by ``AbstractDataSet`` implementations
Expand All @@ -43,12 +48,6 @@ class DatasetError(Exception):
pass


class DataSetError(metaclass=DeprecatedClassMeta):
# pylint: disable=missing-class-docstring, too-few-public-methods

_DeprecatedClassMeta__alias = DatasetError


class DatasetNotFoundError(DatasetError):
"""``DatasetNotFoundError`` raised by ``DataCatalog`` class in case of
trying to use a non-existing data set.
Expand All @@ -57,12 +56,6 @@ class DatasetNotFoundError(DatasetError):
pass


class DataSetNotFoundError(metaclass=DeprecatedClassMeta):
# pylint: disable=missing-class-docstring, too-few-public-methods

_DeprecatedClassMeta__alias = DatasetNotFoundError


class DatasetAlreadyExistsError(DatasetError):
"""``DatasetAlreadyExistsError`` raised by ``DataCatalog`` class in case
of trying to add a data set which already exists in the ``DataCatalog``.
Expand All @@ -71,10 +64,24 @@ class DatasetAlreadyExistsError(DatasetError):
pass


class DataSetAlreadyExistsError(metaclass=DeprecatedClassMeta):
# pylint: disable=missing-class-docstring, too-few-public-methods
_DEPRECATED_ERROR_CLASSES = {
"DataSetError": DatasetError,
"DataSetNotFoundError": DatasetNotFoundError,
"DataSetAlreadyExistsError": DatasetAlreadyExistsError,
}

_DeprecatedClassMeta__alias = DatasetAlreadyExistsError

def __getattr__(name):
if name in _DEPRECATED_ERROR_CLASSES:
alias = _DEPRECATED_ERROR_CLASSES[name]
warnings.warn(
f"{repr(name)} has been renamed to {repr(alias.__name__)}, "
f"and the alias will be removed in Kedro 0.19.0",
DeprecationWarning,
stacklevel=2,
)
return alias
raise AttributeError(f"module {repr(__name__)} has no attribute {repr(name)}")


class VersionNotFoundError(DatasetError):
Expand Down
15 changes: 14 additions & 1 deletion tests/io/test_core.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
from __future__ import annotations

import importlib
from decimal import Decimal
from fractions import Fraction
from pathlib import PurePosixPath
from typing import Any

import pytest

from kedro.io.core import AbstractDataSet, _parse_filepath, get_filepath_str
from kedro.io.core import (
_DEPRECATED_ERROR_CLASSES,
AbstractDataSet,
_parse_filepath,
get_filepath_str,
)

# List sourced from https://docs.python.org/3/library/stdtypes.html#truth-value-testing.
# Excludes None, as None values are not shown in the str representation.
Expand All @@ -27,6 +33,13 @@
]


@pytest.mark.parametrize("module_name", ["kedro.io", "kedro.io.core"])
@pytest.mark.parametrize("class_name", _DEPRECATED_ERROR_CLASSES)
def test_deprecation(module_name, class_name):
with pytest.warns(DeprecationWarning, match=f"{repr(class_name)} has been renamed"):
getattr(importlib.import_module(module_name), class_name)


class MyDataSet(AbstractDataSet):
def __init__(self, var=None):
self.var = var
Expand Down