Improve IPython extension error handling

docs/source/development/commands_reference.md

@@ -491,18 +491,14 @@ To start an IPython shell:
 kedro ipython
 ```
 
-Every time you start or restart a notebook kernel, a startup script (`<project-root>/.ipython/profile_default/startup/00-kedro-init.py`) will add the following variables in scope:
+The [Kedro IPython extension](../tools_integration/ipython.md) will make the following variables available in your IPython or Jupyter session:
 
-- `context`: An instance of `kedro.framework.context.KedroContext` class or custom context class extending `KedroContext` if one was set to `CONTEXT_CLASS` in `settings.py` file (further details of how to use `context` can be found [in the IPython documentation](../tools_integration/ipython.md))
-- `startup_error` (`Exception`)
-- `catalog`
+* `catalog` (type `DataCatalog`): [Data Catalog](../data/data_catalog.md) instance that contains all defined datasets; this is a shortcut for `context.catalog`
+* `context` (type `KedroContext`): Kedro project context that provides access to Kedro's library components
+* `pipelines` (type `Dict[str, Pipeline]`): Pipelines defined in your [pipeline registry](../nodes_and_pipelines/run_a_pipeline.md#run-a-pipeline-by-name)
+* `session` (type `KedroSession`): [Kedro session](../kedro_project_setup/session.md) that orchestrates a pipeline run
 
-To reload these variables at any point in your notebook (e.g. if you updated `catalog.yml`) use the [line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#line-magics) `%reload_kedro`, which can be also used to see the error message if any of the variables above are undefined.
-
-If you get an error message `Module ``<module_name>`` not found. Make sure to install required project dependencies by running ``pip install -r requirements.txt`` first.` when running any of those commands, it indicates that some Jupyter or IPython dependencies are not installed in your environment. To resolve this you will need to do the following:
-
-1. Make sure the corresponding dependency is present in `src/requirements.txt`
-2. Run [`pip install -r src/requirements.txt`](#install-all-package-dependencies) command from your terminal
+To reload these variables (e.g. if you updated `catalog.yml`) use the `%reload_kedro` line magic, which can also be used to see the error message if any of the variables above are undefined. 
 
 ##### Copy tagged cells
 To copy the code from [cells tagged](https://jupyter-notebook.readthedocs.io/en/stable/changelog.html#cell-tags) with a `node` tag into Python files under `src/<package_name>/nodes/` in a Kedro project:

docs/source/tools_integration/ipython.md

@@ -15,7 +15,7 @@ There are reasons why you may want to use a Notebook, although in general, the p
 
 The recommended way to interact with Kedro in IPython and Jupyter is through the Kedro [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/index.html), `kedro.extras.extensions.ipython`. An [IPython extension](https://ipython.readthedocs.io/en/stable/config/extensions/) is an importable Python module that has a couple of special functions to load and unload it.
 
-The Kedro IPython extension launches a [Kedro session](../kedro_project_setup/session.md) and makes available the useful Kedro variables `catalog`, `context`, `pipelines` and `session`. It also provides the `%reload_kedro`  [line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) that reloads these variables (for example, if you need to update `catalog` following changes to your Data Catalog).
+The Kedro IPython extension launches a [Kedro session](../kedro_project_setup/session.md) and makes available the useful Kedro variables `catalog`, `context`, `pipelines` and `session`. It also provides the `%reload_kedro` [line magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html) that reloads these variables (for example, if you need to update `catalog` following changes to your Data Catalog).
 
 The simplest way to make use of the Kedro IPython extension is through the following commands:
 * `kedro ipython`. This launches an IPython shell with the extension already loaded and is equivalent to the command `ipython --ext kedro.extras.extensions.ipython`.
@@ -24,6 +24,10 @@ The simplest way to make use of the Kedro IPython extension is through the follo
 
 Running any of the above from within your Kedro project will make the `catalog`, `context`, `pipelines` and `session` variables immediately accessible to you.
 
+```{note}
+If these variables are not available then Kedro has not been able to load your project. This could be, for example, due to a malformed configuration file or missing dependencies. The full error message is shown on the terminal used to launch `kedro ipython`, `kedro jupyter notebook` or `kedro jupyter lab`. Alternatively, it can be accessed inside the IPython or Jupyter session directly with `%reload_kedro`.
+```
+
 ### Managed Jupyter instances
 
 If the above commands are not available to you (e.g. you work in a managed Jupyter service such as a Databricks Notebook) then equivalent behaviour can be achieved by explicitly loading the Kedro IPython extension with the `%load_ext` line magic:

kedro/extras/extensions/ipython.py

@@ -1,4 +1,4 @@
-# pylint: disable=import-outside-toplevel,global-statement,invalid-name
+# pylint: disable=import-outside-toplevel,global-statement,invalid-name,too-many-locals
 """
 This script creates an IPython extension to load Kedro-related variables in
 local scope.
@@ -8,9 +8,6 @@
 from pathlib import Path
 from typing import Any, Dict
 
-from IPython import get_ipython
-from IPython.core.magic import needs_local_scope, register_line_magic
-
 logger = logging.getLogger(__name__)
 default_project_path = Path.cwd()
 
@@ -39,16 +36,17 @@ def reload_kedro(
 ):
     """Line magic which reloads all Kedro default variables.
     Setting the path will also make it default for subsequent calls.
-
-
     """
+    # If a path is provided, set it as default for subsequent calls
+    from IPython import get_ipython
+    from IPython.core.magic import needs_local_scope, register_line_magic
+
     from kedro.framework.cli import load_entry_points
     from kedro.framework.project import LOGGING  # noqa # pylint:disable=unused-import
     from kedro.framework.project import configure_project, pipelines
     from kedro.framework.session import KedroSession
     from kedro.framework.startup import bootstrap_project
 
-    # If a path is provided, set it as default for subsequent calls
     global default_project_path
     if path:
         default_project_path = Path(path).expanduser().resolve()
@@ -63,7 +61,6 @@ def reload_kedro(
     session = KedroSession.create(
         metadata.package_name, default_project_path, env=env, extra_params=extra_params
     )
-    logger.debug("Loading the context from %s", default_project_path)
     context = session.load_context()
     catalog = context.catalog
 
@@ -95,12 +92,11 @@ def load_ipython_extension(ipython):
 
     default_project_path = _find_kedro_project(Path.cwd())
 
-    try:
-        reload_kedro(default_project_path)
-    except (ImportError, ModuleNotFoundError):
-        logger.error("Kedro appears not to be installed in your current environment.")
-    except Exception:  # pylint: disable=broad-except
+    if default_project_path is None:
         logger.warning(
             "Kedro extension was registered but couldn't find a Kedro project. "
             "Make sure you run '%reload_kedro <path_to_kedro_project>'."
         )
+        return
+
+    reload_kedro(default_project_path)

requirements.txt

@@ -10,7 +10,6 @@ importlib_metadata>=3.6  # The "selectable" entry points were introduced in `imp
 jmespath>=0.9.5, <1.0
 pip-tools~=6.5
 pluggy~=1.0.0
-python-json-logger~=2.0
 PyYAML>=4.2, <7.0
 rich~=12.0
 rope~=0.21.0  # subject to LGPLv3 license

tests/extras/extensions/test_ipython.py

@@ -13,12 +13,7 @@ def project_path(mocker, tmp_path):
 
 
 @pytest.fixture(autouse=True)
-def cleanup_session():
-    yield
-
-
-@pytest.fixture()
-def pipeline_cleanup():
+def cleanup_pipeline():
     yield
     from kedro.framework.project import pipelines
 
@@ -31,7 +26,6 @@ def pipeline_cleanup():
 
 
 class TestLoadKedroObjects:
-    @pytest.mark.usefixtures("pipeline_cleanup")
     def test_load_kedro_objects(
         self, tmp_path, mocker, caplog
     ):  # pylint: disable=too-many-locals
@@ -62,18 +56,18 @@ def my_register_pipeline():
         mocker.patch(
             "kedro.framework.startup.bootstrap_project", return_value=fake_metadata
         )
-        mock_line_magic = mocker.MagicMock()
+        mock_line_magic = mocker.Mock()
         mock_line_magic.__name__ = "abc"
         mocker.patch(
             "kedro.framework.cli.load_entry_points", return_value=[mock_line_magic]
         )
         mock_register_line_magic = mocker.patch(
-            "kedro.extras.extensions.ipython.register_line_magic"
+            "IPython.core.magic.register_line_magic"
         )
         mock_session_create = mocker.patch(
             "kedro.framework.session.KedroSession.create"
         )
-        mock_ipython = mocker.patch("kedro.extras.extensions.ipython.get_ipython")
+        mock_ipython = mocker.patch("IPython.get_ipython")
 
         reload_kedro(kedro_path)
 
@@ -113,18 +107,18 @@ def test_load_kedro_objects_extra_args(self, tmp_path, mocker):
         mocker.patch(
             "kedro.framework.startup.bootstrap_project", return_value=fake_metadata
         )
-        mock_line_magic = mocker.MagicMock()
+        mock_line_magic = mocker.Mock()
         mock_line_magic.__name__ = "abc"
         mocker.patch(
             "kedro.framework.cli.load_entry_points", return_value=[mock_line_magic]
         )
         mock_register_line_magic = mocker.patch(
-            "kedro.extras.extensions.ipython.register_line_magic"
+            "IPython.core.magic.register_line_magic"
         )
         mock_session_create = mocker.patch(
             "kedro.framework.session.KedroSession.create"
         )
-        mock_ipython = mocker.patch("kedro.extras.extensions.ipython.get_ipython")
+        mock_ipython = mocker.patch("IPython.get_ipython")
 
         reload_kedro(tmp_path, env="env1", extra_params={"key": "val"})
 
@@ -141,18 +135,6 @@ def test_load_kedro_objects_extra_args(self, tmp_path, mocker):
         )
         assert mock_register_line_magic.call_count == 1
 
-    def test_load_kedro_objects_not_in_kedro_project(self, tmp_path, mocker):
-        mocker.patch(
-            "kedro.framework.startup._get_project_metadata", side_effect=RuntimeError
-        )
-        mock_ipython = mocker.patch("kedro.extras.extensions.ipython.get_ipython")
-
-        with pytest.raises(RuntimeError):
-            reload_kedro(tmp_path)
-        assert not mock_ipython().called
-        assert not mock_ipython().push.called
-
-    @pytest.mark.usefixtures("pipeline_cleanup")
     def test_load_kedro_objects_no_path(self, tmp_path, caplog, mocker):
         from kedro.extras.extensions.ipython import default_project_path
 
@@ -181,14 +163,14 @@ def my_register_pipeline():
         mocker.patch(
             "kedro.framework.startup.bootstrap_project", return_value=fake_metadata
         )
-        mock_line_magic = mocker.MagicMock()
+        mock_line_magic = mocker.Mock()
         mock_line_magic.__name__ = "abc"
         mocker.patch(
             "kedro.framework.cli.load_entry_points", return_value=[mock_line_magic]
         )
-        mocker.patch("kedro.extras.extensions.ipython.register_line_magic")
+        mocker.patch("IPython.core.magic.register_line_magic")
         mocker.patch("kedro.framework.session.KedroSession.load_context")
-        mocker.patch("kedro.extras.extensions.ipython.get_ipython")
+        mocker.patch("IPython.get_ipython")
 
         reload_kedro()
 
@@ -203,36 +185,36 @@ def my_register_pipeline():
 
 
 class TestLoadIPythonExtension:
-    @pytest.mark.parametrize(
-        "error,expected_log_message,level",
-        [
-            (
-                ImportError,
-                "Kedro appears not to be installed in your current environment.",
-                "ERROR",
-            ),
-            (
-                RuntimeError,
-                "Kedro extension was registered but couldn't find a Kedro project. "
-                "Make sure you run '%reload_kedro <path_to_kedro_project>'.",
-                "WARNING",
-            ),
-        ],
-    )
-    def test_load_extension_not_in_kedro_env_or_project(
-        self, error, expected_log_message, level, mocker, caplog
-    ):
-        mocker.patch("kedro.framework.startup._get_project_metadata", side_effect=error)
-        mock_ipython = mocker.patch("kedro.extras.extensions.ipython.get_ipython")
-
-        load_ipython_extension(mocker.MagicMock())
+    def test_load_extension_not_in_kedro_env_or_project(self, mocker):
+        mocker.patch(
+            "kedro.extras.extensions.ipython.reload_kedro", side_effect=ImportError
+        )
+        mocker.patch(
+            "kedro.extras.extensions.ipython._find_kedro_project",
+            return_value=mocker.Mock(),
+        )
+        mock_ipython = mocker.patch("IPython.get_ipython")
+
+        with pytest.raises(ImportError):
+            load_ipython_extension(mocker.Mock())
+
+        assert not mock_ipython().called
+        assert not mock_ipython().push.called
+
+    def test_load_extension_not_in_kedro_project(self, mocker, caplog):
+        mocker.patch(
+            "kedro.extras.extensions.ipython._find_kedro_project", return_value=None
+        )
+        mock_ipython = mocker.patch("IPython.get_ipython")
+
+        load_ipython_extension(mocker.Mock())
 
         assert not mock_ipython().called
         assert not mock_ipython().push.called
 
-        log_messages = [
-            record.getMessage()
-            for record in caplog.records
-            if record.levelname == level
-        ]
-        assert log_messages == [expected_log_message]
+        log_messages = [record.getMessage() for record in caplog.records]
+        expected_message = (
+            "Kedro extension was registered but couldn't find a Kedro project. "
+            "Make sure you run '%reload_kedro <path_to_kedro_project>'."
+        )
+        assert expected_message in log_messages