Support python 3.11

[sbrugman]

Are there any changes that need to be made, or is it as simple as updating the package information?

depends on #2742

[merelcht]

That's a good question @sbrugman. I don't know on top of my head how much needs to be changed, but we can definitely add this to our backlog and start the work on it.

[merelcht]

The first step is to add a CI build with 3.11 and see if that just works, both on kedro and kedro-plugins. If it turns out to be more involved, we should re-estimate or create a follow up ticket.

[noklam]

Summary
See details in CircleCI Logs

• Tensorflow isn't ready for python3.11 - so we can't add Python3.11 until 0.19.
• Spark 3.11 requires Spark >3.4 https://stackoverflow.com/questions/75048688/picklingerror-could-not-serialize-object-indexerror-tuple-index-out-of-range
• Some more random tests breaking even in the framework side - not sure what's the reason
• holoviews datasets

Going to support this will probably take some effort, I would guess an 8 or more. I stopped here just to timebox the task. I would like to get some comments to see if I need to continue. @merelcht @astrojuanlu

Further questions

1. Do we ever drop DataSet supports?
   Every dataset added is making CI more complex. In reality, no one really uses all of the datasets, but our CI has to take care of the combinations of all dependencies. This confuses pip as some dependencies are old and it hits weird combination, sometimes it just stop CI to run completely and it's difficult to figure out which package is the root cause due to 2nd or even 3rd level dependencies i.e. C depends on B depends on A, and CI complain about A. I can only follow a trial & error pattern and bump one library at a time.

2. How do we handle release for kedro-datasets when some libraries are not ready?
   Is this considered a breaking change and we will bump the major version immediately? Semantic Versioning. It also add more complexity to CI, we will need to skip certain test to run in a particular Python version. Kedro-dataset release process kedro-plugins#405

3. Is there a way to handle (1) better?
   It would be great if the team can share some wisdom, or how to approach this problem systematically. I haven't find a better way to make small changes incrementally. It could be just a quick discussion, if there is no better way it's fine too.

4. How do we handle versions conflict?
   holoviews is also pinned at some 2020 versions for a while, it is conflicting with the newer matplotlib version, as a result we have this pin in our test_requirements.txt
   Is this the right thing to do? We aren't testing the newer version, so in theory matplotlib could break with version >3.6 but we will not find out in our test suite Check if improvements are needed for test_requirements.txt setup #1498 @dee

matplotlib>=3.0.3, <3.4; python_version < '3.10' # 3.4.0 breaks holoviews
matplotlib>=3.5, <3.6; python_version == '3.10'
"matplotlib.MatplotlibWriter": ["matplotlib>=3.0.3, <4.0"]

[merelcht]

• Tensorflow isn't ready for python3.11 - so we can't add Python3.11 until 0.19.

It looks like they added support for python 3.11 yesterday 👀 https://github.com/tensorflow/tensorflow/releases/tag/v2.12.0

[astrojuanlu]

Generally speaking I think it's a pity we can't declare Kedro 3.11-compatible because 1 specific dataset is not - this way we are bound to be as conservative as the most conservative of our dependencies. And I completely agree that (1) is a problem for development, at the moment all the dependencies are needed to install the docs and it's a bit of a pain.

I'm thinking if there is a way to decouple the Kedro CI from kedro-org/kedro-plugins#405, but I don't see an easy path. We'd need to selectively install portions of kedro-datasets depending on the Python version (not the end of the world I suppose, but a bit of CI complexity) and mark our tests so that we can conditionally disable them for certain Python versions.

Rehashing what I said in kedro-org/kedro-plugins#405, I'm wondering if it's time to unbundle kedro-datasets a bit, because it's growing quite a lot (I think about this every time I have to install biopython or some other domain-specific packages). Or at least, move some of the problematic datasets that have huge or niche dependencies to separate packages. That would also make community contributions easier I think.

[merelcht]

Summary See details in CircleCI Logs

• Tensorflow isn't ready for python3.11 - so we can't add Python3.11 until 0.19.
• Spark 3.11 requires Spark >3.4 https://stackoverflow.com/questions/75048688/picklingerror-could-not-serialize-object-indexerror-tuple-index-out-of-range
• Some more random tests breaking even in the framework side - not sure what's the reason
• holoviews datasets

Going to support this will probably take some effort, I would guess an 8 or more. I stopped here just to timebox the task. I would like to get some comments to see if I need to continue. @merelcht @astrojuanlu

I'm fine with pausing for now and creating a new ticket for the actual implementation for another spring. If the only things that break are datasets, could we just ignore these tests for 3.11? I guess we might need to also try the 3.11 for kedro-datasets to see if we can easily support it? Ideally we'd get 3.11 on Kedro in a non-breaking release, but for kedro-datasets it's fine to do a breaking release IMO.

Further questions

1. Do we ever drop DataSet supports?
   Every dataset added is making CI more complex. In reality, no one really uses all of the datasets, but our CI has to take care of the combinations of all dependencies. This confuses pip as some dependencies are old and it hits weird combination, sometimes it just stop CI to run completely and it's difficult to figure out which package is the root cause due to 2nd or even 3rd level dependencies i.e. C depends on B depends on A, and CI complain about A. I can only follow a trial & error pattern and bump one library at a time.

I don't think we should drop support for datasets, but we might need to revisit how we run our datasets tests. Maybe with another setup (tox or something) we could just run every dataset class test separately?

2. How do we handle release for kedro-datasets when some libraries are not ready?
   Is this considered a breaking change and we will bump the major version immediately? Semantic Versioning. It also add more complexity to CI, we will need to skip certain test to run in a particular Python version. Kedro-dataset release process #2433

Do we know yet what is required on datasets to support 3.11? I guess depending on what we need to do it might be a major bump, which would be fine with me. I don't care too much about the CI setup, we're doing it similarly on Kedro already.

4. How do we handle versions conflict?
   holoviews is also pinned at some 2020 versions for a while, it is conflicting with the newer matplotlib version, as a result we have this pin in our test_requirements.txt

We might need to review all the dependencies in datasets and check if we can bump them. I don't think we've reviewed many of them since they were added.

[noklam]

Thank you comments on both! I'll try to address all the comments here.

1. Further unbundle datasets
   My gut feeling is that this creates too many overheads for maintenance, do we start to release them as individual meta-packages? Initially, we briefly discussed the meta-package approach but didn't go with it. Explore how other (meta-)packages handle dependencies #1777 If we have a good idea how this could be implemented we can put Kedro-dataset release process kedro-plugins#405 intol Tech design.

2. Kedro / kedro-datasets 3.11 support
   We can definitely release kedro-datasets 3.11 separately. My main concern is before kedro 3.11 is out it's not too meaningful to release kedro-datasets for 3.11, should we release them at the same time to make it more cohesive? Right now we cannot support 3.11 for kedro unless we solves all the datasets problem because we still have kedro.extras.datasets, so I don't see a way that we can have 3.11 support for kedro without solving all the datasets problem. @merelcht

3. Modify the way we run tests
   In theory, running the tests separately isn't the same as installing all the dependencies (But I am not sure if is one superior to another). Imagine a case when datasets have such dependencies. If we do want to run a test like this, we should start with Check if improvements are needed for test_requirements.txt setup #1498, since there is no point to have a different test_requirements.txt anymore.

# Package A
`library_a > 1.0, <2.0

# Pacakge B 
library_a > 2.0, 3.0

Tests will run properly separately, but not when users want to use both datasets at the same time (which is something we guarantee at the moment)

4. What are needed to support Python 3.11?

Do we know yet what is required on datasets to support 3.11? I guess depending on what we need to do it might be a major bump, which would be fine with me. I don't care too much about the CI setup, we're doing it similarly on Kedro already. @merelcht
I think currently the largest issues are tensorflow, I'll need to run a few more tests since the Python3.11 support was just added last week.

[noklam]

Python3.11 support would be added for Spark 3.4.0. At the moment it's on 3.3.2, the next candidate will be 3.4.0, no schedule yet.

Edited: https://is.spark.released.info/, spark release is coming in 25 days, so we should re-visit it in 2/3 sprints later.
apache/spark#38987

[noklam]

Suggested Plan for release:

1. Fix all datasets issue and release kedro-datasets support for Python 3.11 - (Might need to point to the main branch instead of pypi because we won't have kedro available for Python 3.11 yet - Add Python3.11 support for kedro-datasets kedro-plugins#154
2. Release kedro for Python 3.11 - This ticket
3. Revert the CI to point to pypi (kedro-datasets)

[astrojuanlu]

Just to recap, what is missing here?

In #2417 there was some discussion about tensorflow, but it looks like tensorflow has Python 3.11 support now (tensorflow/tensorflow#58032 closed at the beginning of April). So, I guess we should try and add 3.11 support for the kedro-datasets CI first?

[astrojuanlu]

Step (1) above can't proceed unless we have a version of Kedro that bumps requires-python, otherwise it can't even be installed. I think we might need to rework the plan and release Kedro framework with support for Python 3.11 first, otherwise this might be a bit cumbersome. Wondering if we could do that before 0.19, I get asked about this very often.

[noklam]

@astrojuanlu I agree kedro need to be done before kedro-datasets because kedro-datasets depends on kedro instead of the other way round.

I don't see any problem unless we introduce a breaking change, there was comment that should we delay the Python3.11 support for 0.19.0. But I don't see this tagged as 0.19.0 so I assume we can release as soon as we finish. Cc @merelcht and @yetudada for confirmation.

[astrojuanlu]

First: kedro needs to allow 3.11 before kedro-datasets can start the process. But #2417 got stuck, because our e2e tests install kedro from PyPI:

#2417 (comment)

Second: there have been long conversations about not allowing upper version caps in Requires-Python:

• Blog post https://iscinumpy.dev/post/bound-version-constraints/#pinning-the-python-version-is-special
• Long discussion https://discuss.python.org/t/requires-python-upper-limits/12663?u=astrojuanlu

And third: Kedro does work with Python 3.11 already. This is what I did:

1. Modified pyproject.toml locally to remove the upper cap:

diff --git a/pyproject.toml b/pyproject.toml
index ec06b75d..0421b726 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ authors = [
     {name = "Kedro"}
 ]
 description = "Kedro helps you build production-ready data and analytics pipelines"
-requires-python = ">=3.7, <3.11"
+requires-python = ">=3.7"
 keywords = [
     "pipelines",
     "machine learning",

2. Built a wheel and installed it: python -m build && pip install dist/*.whl
3. Created a new project with the spaceflights starter: kedro new --starter=spaceflights
4. Commented out the kedro* dependencies in src/requirements.txt, and manually installed pandas pyarrow openpyxl
5. kedro run 🏁

This worked perfectly!

So, my suggestion is that we proceed as follows:

1. We release a Kedro version without the upper cap, and we don't ever put it again. This could be the upcoming 0.18.11. We don't tell anyone (not even in the changelog) that technically Kedro can be installed in Python 3.11.
   • Users casually trying Kedro on 3.11 may see installation issues here and there. No worries - Python 3.11 support is by no means official, yet. They will probably arrive to this issue. Not very different from what's happening now.
2. With that version published, we unblock Attempt to bump python version to 3.11  #2417.
3. In parallel (no need to even wait for Attempt to bump python version to 3.11  #2417) we start working on kedro-datasets, kedro-viz, kedro-telemetry, kedro-airflow.
4. In the next release (say 0.18.12) we declare official Python 3.11 support on our changelog (maybe with some exceptions of problematic datasets that we need to skip on CI)
5. In 0.19.0 we drop Python 3.7 (Drop support for Python 3.7 end of year 2023 #2158)

How does that sound?

[noklam]

kedro doesn’t seem to be a problem once I open the upper bound. Maybe it never install kedro from requirements.txt because of this

kedro/features/environment.py

Lines 96 to 113 in 931366f

	def _setup_minimal_env(context):
	kedro_install_venv_dir = _create_new_venv()
	context.kedro_install_venv_dir = kedro_install_venv_dir
	context = _setup_context_with_venv(context, kedro_install_venv_dir)
	call(
	[
	context.python,
	"-m",
	"pip",
	"install",
	"-U",
	"pip>=21.2",
	"setuptools>=65.5.1",
	"wheel",
	],
	env=context.env,
	)
	call([context.python, "-m", "pip", "install", "."], env=context.env)

Even if we release an open bound kedro, we will be still blocked by kedro-telemetry which depends on kedro and don’t have 3.11 yet.

https://app.circleci.com/pipelines/github/kedro-org/kedro/23779/workflows/2d306aed-ed9b-498a-aec4-bb05f5429fb9/jobs/273978

When we release 0.18, we also have a failing e2e test that we bypass manually. #1399

[noklam]

Quick update about the Python 3.11 work. 2 remaining blockers:
Matplotlib seems getting into the way - this shows up after we move to Github Action CI, potentially build issue
@Sajid Alam
is looking at this
We need a kedro-telemetry for Python 3.11

[SajidAlamQB]

🎉 Great news! We've merged the Python 3.11 work. Check out the PR here: #2851. We appreciate your feedback – please don't hesitate to share any thoughts or issues you come across.

[noklam]

This will be coming in 0.18.13, if you can't wait. You can also clone the repository and install from main branch.

[merelcht]

Completed in #2851