Upgrade `Python` 3.11 #1844

bjornjorgensen · 2022-12-08T14:34:33Z

Describe your changes

Upgrade python from 3.10 to 3.11

Release notes

TensorFlow is now set to be installed using pip instead of mamba.

Issue ticket if applicable

Checklist (especially for first-time contributors)

I have performed a self-review of my code
If it is a core feature, I have added thorough tests
I will try not to use force-push to make the review process easier for reviewers
I have updated the documentation for significant changes

bjornjorgensen · 2022-12-08T16:47:23Z

[3/7] RUN mamba install --quiet --yes 'altair' 'beautifulsoup4' 'bokeh' 'bottleneck' 'cloudpickle' 'conda-forge::blas=*=openblas' 'cython' 'dask' 'dill' 'h5py' 'ipympl' 'ipywidgets' 'matplotlib-base' 'numba' 'numexpr' 'openpyxl' 'pandas' 'patsy' 'protobuf' 'pytables' 'scikit-image' 'scikit-learn' 'scipy' 'seaborn' 'sqlalchemy' 'statsmodels' 'sympy' 'widgetsnbextension' 'xlrd' && mamba clean --all -f -y && fix-permissions "/opt/conda" && fix-permissions "/home/jovyan"
#6 sha256:ac03e34fe6e321c9e69e2e65b043b1b901e58cafc210ede3cd3df4959ad7cf45
#6 32.22 Could not solve for environment specs
#6 32.22 Encountered problems while solving:
#6 32.22 - nothing provides openssl >=1.0.2p,<1.0.3a needed by python-3.6.7-hd21baee_1002
#6 32.22
#6 32.22 The environment can't be solved, aborting the operation
#6 32.22

[3/7] RUN mamba install --quiet --yes 'altair' 'beautifulsoup4' 'bokeh' 'bottleneck' 'cloudpickle' 'conda-forge::blas=*=openblas' 'cython' 'dask' 'dill' 'h5py' 'ipympl' 'ipywidgets' 'matplotlib-base' 'numba' 'numexpr' 'openpyxl' 'pandas' 'patsy' 'protobuf' 'pytables' 'scikit-image' 'scikit-learn' 'scipy' 'seaborn' 'sqlalchemy' 'statsmodels' 'sympy' 'widgetsnbextension' 'xlrd' && mamba clean --all -f -y && fix-permissions "/opt/conda" && fix-permissions "/home/jovyan"
#6 sha256:481075b6d100da5c3b645b3cd2f6cb3b7edd0aee0c3749d4174a2daa1c3eaf2c
#6 77.07 Could not solve for environment specs
#6 77.07 Encountered problems while solving:
#6 77.07 - package numba-0.53.0-py39h4292f66_0 requires python >=3.9,<3.10.0a0, but none of the providers can be installed
#6 77.07
#6 77.07 The environment can't be solved, aborting the operation
#6 77.07

bjornjorgensen · 2022-12-08T16:52:48Z

numba numba/numba#8304

bjornjorgensen · 2022-12-09T17:06:33Z

I think we should wait for spark 3.4 before we upgrade
apache/spark#38987
Spark 3.4 will be out in jan / feb.

mathbunnyru · 2022-12-09T19:13:04Z

That's a bit unfortunate, but I think you're right and we will have to wait until both numba and spark work fine?
Do other packages work properly and have you tried removing/commenting numba and pyspark and building/testing everything else?

bjornjorgensen · 2022-12-09T19:16:24Z

I can test that when #1845 is merged.

mathbunnyru · 2022-12-09T19:17:25Z

I can test that when #1845 is merged.

Haha, I guess my PR came right on time 😂

benz0li · 2022-12-10T10:29:38Z

Do other packages work properly and have you tried removing/commenting numba and pyspark and building/testing everything else?

@mathbunnyru Regarding scipy-notebook: The current release of pytables does not support Python 3.11.
ℹ️ See also #1812 (comment).

parmentelat · 2023-01-19T14:38:33Z

@ benz0li

Regarding scipy-notebook: The current release of pytables does not support Python 3.11.

not sure if this is still current or not, the tables package seems to be pip-installable on 3.11 no ?

benz0li · 2023-01-19T14:51:05Z

not sure if this is still current or not, the tables package seems to be pip-installable on 3.11 no ?

@parmentelat See #1812 (comment). There is only one issue remaining: numba/numba#8304.

mathbunnyru · 2023-01-20T09:00:59Z

I merged main to this PR and removed numba to make sure everything else works as expected.

bjornjorgensen · 2023-01-20T10:01:19Z

tensorflow doesn't support python 3.11 https://stackoverflow.com/questions/74556733/tensorflow-support-for-python3-11

mathbunnyru · 2023-01-20T10:04:10Z

Here is a corresponding issue: tensorflow/tensorflow#58032

mathbunnyru · 2023-01-20T10:05:25Z

Also test_nbconvert fails for all-spark-notebook, so we will probably have to take a look there as well.

bjornjorgensen · 2023-01-20T17:06:25Z

Apache spark will have a rc1 next week. Spark 3.4 have support for python 3.11 I can make a test run with Spark 3.4 rc1 and python 3.11

bjornjorgensen · 2023-02-17T21:56:13Z

Spark 3.3.2 does not have support for python 3.11.
Spark 3.4 will soon have a RC1 witch have support for python 3.11

mathbunnyru · 2023-04-04T07:39:22Z

Numba and TensorFlow issues about python 3.11 were closed.
Now we need to wait until the recipes in conda-forge are updated.

bjornjorgensen · 2023-04-04T07:54:51Z

great, but Apache spark 3.4 will have another RC. So I think we need to wait one or two weeks.

benz0li · 2023-04-04T09:06:18Z

Numba and TensorFlow issues about python 3.11 were closed.

See the Release Candidate checklist for Numba 0.57.0rc1 at numba/numba#8841.

mathbunnyru · 2023-05-19T06:26:42Z

Numba v0.57.0 was released yesterday:

A new release of Numba adding support for Python 3.11, NumPy 1.24, and upgrading to LLVM 14. For details, please see the change log at https://numba.readthedocs.io/en/stable/release-notes.html#version-0-57-0-1-may-2023.

Great news 👍 We will have to wait for conda-forge release, hope it won't take too long. conda-forge/numba-feedstock#115

This PR has just been merged 🎉

bjornjorgensen · 2023-05-19T11:38:30Z

Tensorflow version 2.12.0 have support for python 3.11 https://github.com/tensorflow/tensorflow/releases/tag/v2.12.0
But the "Do not install TensorFlow with conda. It may not have the latest stable version. pip is recommended since TensorFlow is only officially released to PyPI."
https://www.tensorflow.org/install/pip

bjornjorgensen · 2023-05-19T11:41:01Z

should we mamba insall pip in https://github.com/jupyter/docker-stacks/blob/main/tensorflow-notebook/Dockerfile
and then pip install tensorflow?

https://anaconda.org/anaconda/pip

mathbunnyru · 2023-05-19T11:48:58Z

In my experience, it's always better to use conda/mamba and not mix them with pip.
There is active work on the new TensorFlow here: conda-forge/tensorflow-feedstock#316
So, let's wait for a while, I think they might release new version quite soon.

mathbunnyru · 2023-05-31T13:52:44Z

@bjornjorgensen sorry for triggering CI so many times and making GitHub spam you - it was a really nice opportunity to test our new aarch64 runners 😂

mathbunnyru · 2023-05-31T13:55:12Z

Currently, I think we can proceed with pypi tensorflow.
Let's give it at least a try.

I also suppose, we will be able to build aarch64 tensorflow-notebook, but I suggest to enable it in a separate PR after merging this one.

bjornjorgensen · 2023-05-31T14:10:46Z

ok, so you will merge this one now and start a new PR where we conda install pip && pip install tensorflow==2.12.*
and then we probely have to do something like

Ubuntu 22.04
In Ubuntu 22.04, you may encounter the following error:

Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice.
...
Couldn't invoke ptxas --version
...
InternalError: libdevice not found at ./libdevice.10.bc [Op:__some_op]
To fix this error, you will need to run the following commands.

Install NVCC

conda install -c nvidia cuda-nvcc=11.3.58

Configure the XLA cuda directory

mkdir -p $CONDA_PREFIX/etc/conda/activate.d
printf 'export XLA_FLAGS=--xla_gpu_cuda_data_dir=$CONDA_PREFIX/lib/\n' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

Copy libdevice file to the required path

mkdir -p $CONDA_PREFIX/lib/nvvm/libdevice
cp $CONDA_PREFIX/lib/libdevice.10.bc $CONDA_PREFIX/lib/nvvm/libdevice/

mathbunnyru · 2023-05-31T14:14:18Z

so you will merge this one now

No, please, let's try use pip in this PR and if it works, I will merge it.
About the errors - are you sure we will have these errors?
I suppose we're using non-coda version, so we shouldn't see there errors?

bjornjorgensen · 2023-05-31T16:06:06Z

No, please, let's try use pip in this PR and if it works, I will merge it. About the errors - are you sure we will have these errors? I suppose we're using non-coda version, so we shouldn't see there errors?

I try to follow https://stackoverflow.com/questions/41060382/using-pip-to-install-packages-to-anaconda-environment
so now I hope that we install pip in conda and use pip witch are in conda to install tensorflow.

bjornjorgensen · 2023-05-31T17:00:10Z

a local test

# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
ARG OWNER=jupyter
ARG BASE_CONTAINER=$OWNER/scipy-notebook
FROM $BASE_CONTAINER

LABEL maintainer="Jupyter Project <[email protected]>"

# Fix: https://github.com/hadolint/hadolint/wiki/DL4006
# Fix: https://github.com/koalaman/shellcheck/wiki/SC3014
SHELL ["/bin/bash", "-o", "pipefail", "-c"]



ARG PYTHON_VERSION=3.11

# Setup work directory for backward-compatibility
#RUN mkdir "/home/${NB_USER}/work" && \
#    fix-permissions "/home/${NB_USER}"
# Download and install Micromamba, and initialize Conda prefix.
#   <https://github.com/mamba-org/mamba#micromamba>
#   Similar projects using Micromamba:
#     - Micromamba-Docker: <https://github.com/mamba-org/micromamba-docker>
#     - repo2docker: <https://github.com/jupyterhub/repo2docker>
# Install Python, Mamba and jupyter_core
# Cleanup temporary files and remove Micromamba
# Correct permissions
# Do all this in a single RUN command to avoid duplicating all of the
# files across image layers when the permissions change
RUN rm -R "${CONDA_DIR}/conda-meta/pinned"
COPY --chown="${NB_UID}:${NB_GID}" initial-condarc "${CONDA_DIR}/.condarc"
WORKDIR /tmp
RUN set -x && \
    arch=$(uname -m) && \
    if [ "${arch}" = "x86_64" ]; then \
        # Should be simpler, see <https://github.com/mamba-org/mamba/issues/1437>
        arch="64"; \
    fi && \
    wget -qO /tmp/micromamba.tar.bz2 \
        "https://micromamba.snakepit.net/api/micromamba/linux-${arch}/latest" && \
    tar -xvjf /tmp/micromamba.tar.bz2 --strip-components=1 bin/micromamba && \
    rm /tmp/micromamba.tar.bz2 && \
    PYTHON_SPECIFIER="python=${PYTHON_VERSION}" && \
    if [[ "${PYTHON_VERSION}" == "default" ]]; then PYTHON_SPECIFIER="python"; fi && \
    # Install the packages
    ./micromamba install \
        --root-prefix="${CONDA_DIR}" \
        --prefix="${CONDA_DIR}" \
        --yes \
        "${PYTHON_SPECIFIER}" \
        'mamba' \
        'jupyter_core' && \
    rm micromamba && \
    # Pin major.minor version of python
    mamba list python | grep '^python ' | tr -s ' ' | cut -d ' ' -f 1,2 >> "${CONDA_DIR}/conda-meta/pinned" && \
    mamba clean --all -f -y && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"


# Install Tensorflow with pip
RUN mamba install --yes \
    'pip' && \
    mamba clean --all -f -y && \
    pip install tensorflow && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}"

docker build --pull --rm -f "Dockerfile" -t testdocker:latest "."


Sending build context to Docker daemon   5.12kB
Step 1/11 : ARG OWNER=jupyter
Step 2/11 : ARG BASE_CONTAINER=$OWNER/scipy-notebook
Step 3/11 : FROM $BASE_CONTAINER
latest: Pulling from jupyter/scipy-notebook
..........................................


Successfully installed absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.1 flatbuffers-23.5.26 gast-0.4.0 google-auth-2.19.0 google-auth-oauthlib-1.0.0 google-pasta-0.2.0 grpcio-1.54.2 jax-0.4.10 keras-2.12.0 libclang-16.0.0 markdown-3.4.3 ml-dtypes-0.1.0 numpy-1.23.5 opt-einsum-3.3.0 pyasn1-0.5.0 pyasn1-modules-0.3.0 requests-oauthlib-1.3.1 rsa-4.9 tensorboard-2.12.3 tensorboard-data-server-0.7.0 tensorflow-2.12.0 tensorflow-estimator-2.12.0 tensorflow-io-gcs-filesystem-0.32.0 termcolor-2.3.0 urllib3-1.26.16 werkzeug-2.3.4 wrapt-1.14.1
Removing intermediate container cc6653edfa0f
 ---> ab70dc9c1c01
Successfully built ab70dc9c1c01
Successfully tagged testdocker:latest

tensorflow-notebook/Dockerfile

mathbunnyru

I will merge when the tests pass.

bjornjorgensen · 2023-05-31T17:34:34Z

ok, should we update the PR description?

mathbunnyru · 2023-05-31T17:38:28Z

ok, should we update the PR description?

I think it's ok now.

* Update Dockerfile * Update test_python.py * [TMP] Do not install numba * test with spark 3.4 rc2 * Update Dockerfile * install tf with pip in conda * remove mamba install pip * Update Dockerfile --------- Co-authored-by: Ayaz Salikhov <[email protected]> Co-authored-by: Ayaz Salikhov <[email protected]>

bjornjorgensen added 2 commits December 8, 2022 15:31

Update Dockerfile

fa432cf

Update test_python.py

89975ed

bjornjorgensen changed the title ~~Patch 1~~ Upgrade Python 3.11 Dec 8, 2022

mathbunnyru closed this Jan 20, 2023

mathbunnyru reopened this Jan 20, 2023

mathbunnyru added 2 commits January 20, 2023 12:57

Merge branch 'main' into HEAD

4715067

[TMP] Do not install numba

505fa67

benz0li mentioned this pull request Feb 14, 2023

[Python 3.11] - Debugger warning: It seems that frozen modules are being used #1861

Closed

1 task

Merge branch 'main' into HEAD

9f2dcc8

mathbunnyru and others added 2 commits March 7, 2023 12:25

Merge branch 'main' into patch-1

c585b58

test with spark 3.4 rc2

2b36986

Merge branch 'main' into patch-1

9c1a8db

mathbunnyru added 2 commits May 19, 2023 10:25

Merge branch 'main' into patch-1

bbb9502

Update Dockerfile

f9b8b3f

mathbunnyru added 3 commits May 25, 2023 18:16

Merge branch 'main' into patch-1

6d78bba

Merge branch 'main' into patch-1

679dca3

Merge branch 'main' into patch-1

b1f8cb0

mathbunnyru mentioned this pull request May 31, 2023

[ENH] - Python 3.11 compatibility #1910

Closed

Merge branch 'main' into patch-1

e73f956

install tf with pip in conda

e6aedc7

mathbunnyru reviewed May 31, 2023

View reviewed changes

tensorflow-notebook/Dockerfile Outdated Show resolved Hide resolved

mathbunnyru reviewed May 31, 2023

View reviewed changes

tensorflow-notebook/Dockerfile Outdated Show resolved Hide resolved

remove mamba install pip

9e46992

bjornjorgensen commented May 31, 2023

View reviewed changes

tensorflow-notebook/Dockerfile Show resolved Hide resolved

Merge branch 'main' into patch-1

3d8ad83

mathbunnyru approved these changes May 31, 2023

View reviewed changes

Update Dockerfile

f70336b

mathbunnyru merged commit 5bcf210 into jupyter:main Jun 1, 2023

bjornjorgensen mentioned this pull request Mar 10, 2024

Upgrade to Python 3.12 #2072

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade `Python` 3.11 #1844

Upgrade `Python` 3.11 #1844

bjornjorgensen commented Dec 8, 2022 •

edited

Loading

bjornjorgensen commented Dec 8, 2022

bjornjorgensen commented Dec 8, 2022

bjornjorgensen commented Dec 9, 2022

mathbunnyru commented Dec 9, 2022

bjornjorgensen commented Dec 9, 2022

mathbunnyru commented Dec 9, 2022

benz0li commented Dec 10, 2022

parmentelat commented Jan 19, 2023

benz0li commented Jan 19, 2023

mathbunnyru commented Jan 20, 2023 •

edited

Loading

bjornjorgensen commented Jan 20, 2023

mathbunnyru commented Jan 20, 2023

mathbunnyru commented Jan 20, 2023 •

edited

Loading

bjornjorgensen commented Jan 20, 2023

bjornjorgensen commented Feb 17, 2023

mathbunnyru commented Apr 4, 2023

bjornjorgensen commented Apr 4, 2023

benz0li commented Apr 4, 2023

mathbunnyru commented May 19, 2023 •

edited

Loading

bjornjorgensen commented May 19, 2023

bjornjorgensen commented May 19, 2023 •

edited

Loading

mathbunnyru commented May 19, 2023

mathbunnyru commented May 31, 2023

mathbunnyru commented May 31, 2023 •

edited

Loading

bjornjorgensen commented May 31, 2023

mathbunnyru commented May 31, 2023

bjornjorgensen commented May 31, 2023

bjornjorgensen commented May 31, 2023

mathbunnyru left a comment

bjornjorgensen commented May 31, 2023

mathbunnyru commented May 31, 2023

Upgrade Python 3.11 #1844

Upgrade Python 3.11 #1844

Conversation

bjornjorgensen commented Dec 8, 2022 • edited Loading

Describe your changes

Issue ticket if applicable

Checklist (especially for first-time contributors)

bjornjorgensen commented Dec 8, 2022

bjornjorgensen commented Dec 8, 2022

bjornjorgensen commented Dec 9, 2022

mathbunnyru commented Dec 9, 2022

bjornjorgensen commented Dec 9, 2022

mathbunnyru commented Dec 9, 2022

benz0li commented Dec 10, 2022

parmentelat commented Jan 19, 2023

benz0li commented Jan 19, 2023

mathbunnyru commented Jan 20, 2023 • edited Loading

bjornjorgensen commented Jan 20, 2023

mathbunnyru commented Jan 20, 2023

mathbunnyru commented Jan 20, 2023 • edited Loading

bjornjorgensen commented Jan 20, 2023

bjornjorgensen commented Feb 17, 2023

mathbunnyru commented Apr 4, 2023

bjornjorgensen commented Apr 4, 2023

benz0li commented Apr 4, 2023

mathbunnyru commented May 19, 2023 • edited Loading

bjornjorgensen commented May 19, 2023

bjornjorgensen commented May 19, 2023 • edited Loading

mathbunnyru commented May 19, 2023

mathbunnyru commented May 31, 2023

mathbunnyru commented May 31, 2023 • edited Loading

bjornjorgensen commented May 31, 2023

Install NVCC

Configure the XLA cuda directory

Copy libdevice file to the required path

mathbunnyru commented May 31, 2023

bjornjorgensen commented May 31, 2023

bjornjorgensen commented May 31, 2023

mathbunnyru left a comment

Choose a reason for hiding this comment

bjornjorgensen commented May 31, 2023

mathbunnyru commented May 31, 2023

Upgrade `Python` 3.11 #1844

Upgrade `Python` 3.11 #1844

bjornjorgensen commented Dec 8, 2022 •

edited

Loading

mathbunnyru commented Jan 20, 2023 •

edited

Loading

mathbunnyru commented Jan 20, 2023 •

edited

Loading

mathbunnyru commented May 19, 2023 •

edited

Loading

bjornjorgensen commented May 19, 2023 •

edited

Loading

mathbunnyru commented May 31, 2023 •

edited

Loading