diff --git a/RELEASE.md b/RELEASE.md index dcbd33bed6..6ae6981e90 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -15,6 +15,7 @@ ## Bug fixes and other changes * Use default `False` value for rich logging `set_locals`, to make sure credentials and other sensitive data isn't shown in logs. +* The Kedro IPython extension now surfaces errors when it cannot load a Kedro project. ## Upcoming deprecations for Kedro 0.19.0 diff --git a/docs/source/development/commands_reference.md b/docs/source/development/commands_reference.md index be660742c4..6ee404f642 100644 --- a/docs/source/development/commands_reference.md +++ b/docs/source/development/commands_reference.md @@ -491,18 +491,14 @@ To start an IPython shell: kedro ipython ``` -Every time you start or restart a notebook kernel, a startup script (`/.ipython/profile_default/startup/00-kedro-init.py`) will add the following variables in scope: +The [Kedro IPython extension](../tools_integration/ipython.md) will make the following variables available in your IPython or Jupyter session: -- `context`: An instance of `kedro.framework.context.KedroContext` class or custom context class extending `KedroContext` if one was set to `CONTEXT_CLASS` in `settings.py` file (further details of how to use `context` can be found [in the IPython documentation](../tools_integration/ipython.md)) -- `startup_error` (`Exception`) -- `catalog` +* `catalog` (type `DataCatalog`): [Data Catalog](../data/data_catalog.md) instance that contains all defined datasets; this is a shortcut for `context.catalog` +* `context` (type `KedroContext`): Kedro project context that provides access to Kedro's library components +* `pipelines` (type `Dict[str, Pipeline]`): Pipelines defined in your [pipeline registry](../nodes_and_pipelines/run_a_pipeline.md#run-a-pipeline-by-name) +* `session` (type `KedroSession`): [Kedro session](../kedro_project_setup/session.md) that orchestrates a pipeline run -To reload these variables at any point in your notebook (e.g. if you updated `catalog.yml`) use the [line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#line-magics) `%reload_kedro`, which can be also used to see the error message if any of the variables above are undefined. - -If you get an error message `Module ```` not found. Make sure to install required project dependencies by running ``pip install -r requirements.txt`` first.` when running any of those commands, it indicates that some Jupyter or IPython dependencies are not installed in your environment. To resolve this you will need to do the following: - -1. Make sure the corresponding dependency is present in `src/requirements.txt` -2. Run [`pip install -r src/requirements.txt`](#install-all-package-dependencies) command from your terminal +To reload these variables (e.g. if you updated `catalog.yml`) use the `%reload_kedro` line magic, which can also be used to see the error message if any of the variables above are undefined. ##### Copy tagged cells To copy the code from [cells tagged](https://jupyter-notebook.readthedocs.io/en/stable/changelog.html#cell-tags) with a `node` tag into Python files under `src//nodes/` in a Kedro project: diff --git a/docs/source/tools_integration/ipython.md b/docs/source/tools_integration/ipython.md index 7e129dbc04..baa94d0ce7 100644 --- a/docs/source/tools_integration/ipython.md +++ b/docs/source/tools_integration/ipython.md @@ -15,7 +15,7 @@ There are reasons why you may want to use a Notebook, although in general, the p The recommended way to interact with Kedro in IPython and Jupyter is through the Kedro [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/index.html), `kedro.extras.extensions.ipython`. An [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/) is an importable Python module that has a couple of special functions to load and unload it. -The Kedro IPython extension launches a [Kedro session](../kedro_project_setup/session.md) and makes available the useful Kedro variables `catalog`, `context`, `pipelines` and `session`. It also provides the `%reload_kedro` [line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) that reloads these variables (for example, if you need to update `catalog` following changes to your Data Catalog). +The Kedro IPython extension launches a [Kedro session](../kedro_project_setup/session.md) and makes available the useful Kedro variables `catalog`, `context`, `pipelines` and `session`. It also provides the `%reload_kedro` [line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) that reloads these variables (for example, if you need to update `catalog` following changes to your Data Catalog). The simplest way to make use of the Kedro IPython extension is through the following commands: * `kedro ipython`. This launches an IPython shell with the extension already loaded and is equivalent to the command `ipython --ext kedro.extras.extensions.ipython`. @@ -24,6 +24,10 @@ The simplest way to make use of the Kedro IPython extension is through the follo Running any of the above from within your Kedro project will make the `catalog`, `context`, `pipelines` and `session` variables immediately accessible to you. +```{note} +If these variables are not available then Kedro has not been able to load your project. This could be, for example, due to a malformed configuration file or missing dependencies. The full error message is shown on the terminal used to launch `kedro ipython`, `kedro jupyter notebook` or `kedro jupyter lab`. Alternatively, it can be accessed inside the IPython or Jupyter session directly with `%reload_kedro`. +``` + ### Managed Jupyter instances If the above commands are not available to you (e.g. you work in a managed Jupyter service such as a Databricks Notebook) then equivalent behaviour can be achieved by explicitly loading the Kedro IPython extension with the `%load_ext` line magic: diff --git a/kedro/extras/extensions/ipython.py b/kedro/extras/extensions/ipython.py index 406617e094..4e4d9e835d 100644 --- a/kedro/extras/extensions/ipython.py +++ b/kedro/extras/extensions/ipython.py @@ -1,4 +1,4 @@ -# pylint: disable=import-outside-toplevel,global-statement,invalid-name +# pylint: disable=import-outside-toplevel,global-statement,invalid-name,too-many-locals """ This script creates an IPython extension to load Kedro-related variables in local scope. @@ -8,9 +8,6 @@ from pathlib import Path from typing import Any, Dict -from IPython import get_ipython -from IPython.core.magic import needs_local_scope, register_line_magic - logger = logging.getLogger(__name__) default_project_path = Path.cwd() @@ -39,9 +36,10 @@ def reload_kedro( ): """Line magic which reloads all Kedro default variables. Setting the path will also make it default for subsequent calls. - - """ + from IPython import get_ipython + from IPython.core.magic import needs_local_scope, register_line_magic + from kedro.framework.cli import load_entry_points from kedro.framework.project import LOGGING # noqa # pylint:disable=unused-import from kedro.framework.project import configure_project, pipelines @@ -63,7 +61,6 @@ def reload_kedro( session = KedroSession.create( metadata.package_name, default_project_path, env=env, extra_params=extra_params ) - logger.debug("Loading the context from %s", default_project_path) context = session.load_context() catalog = context.catalog @@ -95,12 +92,11 @@ def load_ipython_extension(ipython): default_project_path = _find_kedro_project(Path.cwd()) - try: - reload_kedro(default_project_path) - except (ImportError, ModuleNotFoundError): - logger.error("Kedro appears not to be installed in your current environment.") - except Exception: # pylint: disable=broad-except + if default_project_path is None: logger.warning( "Kedro extension was registered but couldn't find a Kedro project. " "Make sure you run '%reload_kedro '." ) + return + + reload_kedro(default_project_path) diff --git a/requirements.txt b/requirements.txt index e50d7d1c37..33d66b6f0a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,7 +10,6 @@ importlib_metadata>=3.6 # The "selectable" entry points were introduced in `imp jmespath>=0.9.5, <1.0 pip-tools~=6.5 pluggy~=1.0.0 -python-json-logger~=2.0 PyYAML>=4.2, <7.0 rich~=12.0 rope~=0.21.0 # subject to LGPLv3 license diff --git a/tests/extras/extensions/test_ipython.py b/tests/extras/extensions/test_ipython.py index 218a2e2089..6d3f9f9062 100644 --- a/tests/extras/extensions/test_ipython.py +++ b/tests/extras/extensions/test_ipython.py @@ -13,12 +13,7 @@ def project_path(mocker, tmp_path): @pytest.fixture(autouse=True) -def cleanup_session(): - yield - - -@pytest.fixture() -def pipeline_cleanup(): +def cleanup_pipeline(): yield from kedro.framework.project import pipelines @@ -31,7 +26,6 @@ def pipeline_cleanup(): class TestLoadKedroObjects: - @pytest.mark.usefixtures("pipeline_cleanup") def test_load_kedro_objects( self, tmp_path, mocker, caplog ): # pylint: disable=too-many-locals @@ -62,18 +56,18 @@ def my_register_pipeline(): mocker.patch( "kedro.framework.startup.bootstrap_project", return_value=fake_metadata ) - mock_line_magic = mocker.MagicMock() + mock_line_magic = mocker.Mock() mock_line_magic.__name__ = "abc" mocker.patch( "kedro.framework.cli.load_entry_points", return_value=[mock_line_magic] ) mock_register_line_magic = mocker.patch( - "kedro.extras.extensions.ipython.register_line_magic" + "IPython.core.magic.register_line_magic" ) mock_session_create = mocker.patch( "kedro.framework.session.KedroSession.create" ) - mock_ipython = mocker.patch("kedro.extras.extensions.ipython.get_ipython") + mock_ipython = mocker.patch("IPython.get_ipython") reload_kedro(kedro_path) @@ -113,18 +107,18 @@ def test_load_kedro_objects_extra_args(self, tmp_path, mocker): mocker.patch( "kedro.framework.startup.bootstrap_project", return_value=fake_metadata ) - mock_line_magic = mocker.MagicMock() + mock_line_magic = mocker.Mock() mock_line_magic.__name__ = "abc" mocker.patch( "kedro.framework.cli.load_entry_points", return_value=[mock_line_magic] ) mock_register_line_magic = mocker.patch( - "kedro.extras.extensions.ipython.register_line_magic" + "IPython.core.magic.register_line_magic" ) mock_session_create = mocker.patch( "kedro.framework.session.KedroSession.create" ) - mock_ipython = mocker.patch("kedro.extras.extensions.ipython.get_ipython") + mock_ipython = mocker.patch("IPython.get_ipython") reload_kedro(tmp_path, env="env1", extra_params={"key": "val"}) @@ -141,18 +135,6 @@ def test_load_kedro_objects_extra_args(self, tmp_path, mocker): ) assert mock_register_line_magic.call_count == 1 - def test_load_kedro_objects_not_in_kedro_project(self, tmp_path, mocker): - mocker.patch( - "kedro.framework.startup._get_project_metadata", side_effect=RuntimeError - ) - mock_ipython = mocker.patch("kedro.extras.extensions.ipython.get_ipython") - - with pytest.raises(RuntimeError): - reload_kedro(tmp_path) - assert not mock_ipython().called - assert not mock_ipython().push.called - - @pytest.mark.usefixtures("pipeline_cleanup") def test_load_kedro_objects_no_path(self, tmp_path, caplog, mocker): from kedro.extras.extensions.ipython import default_project_path @@ -181,14 +163,14 @@ def my_register_pipeline(): mocker.patch( "kedro.framework.startup.bootstrap_project", return_value=fake_metadata ) - mock_line_magic = mocker.MagicMock() + mock_line_magic = mocker.Mock() mock_line_magic.__name__ = "abc" mocker.patch( "kedro.framework.cli.load_entry_points", return_value=[mock_line_magic] ) - mocker.patch("kedro.extras.extensions.ipython.register_line_magic") + mocker.patch("IPython.core.magic.register_line_magic") mocker.patch("kedro.framework.session.KedroSession.load_context") - mocker.patch("kedro.extras.extensions.ipython.get_ipython") + mocker.patch("IPython.get_ipython") reload_kedro() @@ -203,36 +185,36 @@ def my_register_pipeline(): class TestLoadIPythonExtension: - @pytest.mark.parametrize( - "error,expected_log_message,level", - [ - ( - ImportError, - "Kedro appears not to be installed in your current environment.", - "ERROR", - ), - ( - RuntimeError, - "Kedro extension was registered but couldn't find a Kedro project. " - "Make sure you run '%reload_kedro '.", - "WARNING", - ), - ], - ) - def test_load_extension_not_in_kedro_env_or_project( - self, error, expected_log_message, level, mocker, caplog - ): - mocker.patch("kedro.framework.startup._get_project_metadata", side_effect=error) - mock_ipython = mocker.patch("kedro.extras.extensions.ipython.get_ipython") - - load_ipython_extension(mocker.MagicMock()) + def test_load_extension_missing_dependency(self, mocker): + mocker.patch( + "kedro.extras.extensions.ipython.reload_kedro", side_effect=ImportError + ) + mocker.patch( + "kedro.extras.extensions.ipython._find_kedro_project", + return_value=mocker.Mock(), + ) + mock_ipython = mocker.patch("IPython.get_ipython") + + with pytest.raises(ImportError): + load_ipython_extension(mocker.Mock()) + + assert not mock_ipython().called + assert not mock_ipython().push.called + + def test_load_extension_not_in_kedro_project(self, mocker, caplog): + mocker.patch( + "kedro.extras.extensions.ipython._find_kedro_project", return_value=None + ) + mock_ipython = mocker.patch("IPython.get_ipython") + + load_ipython_extension(mocker.Mock()) assert not mock_ipython().called assert not mock_ipython().push.called - log_messages = [ - record.getMessage() - for record in caplog.records - if record.levelname == level - ] - assert log_messages == [expected_log_message] + log_messages = [record.getMessage() for record in caplog.records] + expected_message = ( + "Kedro extension was registered but couldn't find a Kedro project. " + "Make sure you run '%reload_kedro '." + ) + assert expected_message in log_messages