Re-evaluate pandas and Python versions

[eccabay]

With Pandas version 2.1.0 released on August 30th, the package no longer supports Python versions below 3.9.

We missed this for a long time because the majority of our testing (including all tests that run for PRs, as well as the latest dependency checker) only runs on Python 3.8. This was only discovered now because the pandas upgrade happened on our nightly 3.9 tests after sktime upgraded to version 0.23.0.

Our current plan is to pin pandas under version 2.1.0, but this is not a good long term solution. This issue tracks the discussion around what to do with our Python and pandas versioning in the longer term, as this will only become more of a problem as time goes on and Python 3.8 becomes more out of date.

[eccabay]

The same issue has come up with the networkx package bumping their minimum version to Python 3.9, with the version bumping again to 3.10 in the next release. We have pinned networkx in the meantime, see #4350

[eccabay]

Scipy has not supported < Python 3.9 since v1.10, their newest release of v1.12 has breaking changes with evalml.

[eccabay]

Dependencies which have dropped support for Python 3.8 at this point (3/22/24) are:

• pandas (dropped 2.1.0, currently on 2.2.1)
• networkx (dropped 3.2.0, currently on 3.2.1)
• scipy (dropped 1.11.0, currently on 1.12.0)
• scikit-learn (dropped 1.4.0, currently on 1.4.1.post1)
• numpy (dropped 1.25.0, currently on 1.26.4)
• dask (dropped 2023.5.1, currently on 2024.3.1)
• distributed (dropped 2023.5.1, currently on 2024.3.1)
• matplotlib (dropped 3.8.0, currently on 3.8.3)
• shap (dropped 0.45.0, currently on 0.45.0)
• featuretools (dropped 1.29.0, currently on 1.30.0)
• woodwork (dropped 0.28.8, currently on 0.29.0)

[eccabay]

After the merge of #4414, we have unpinned a number of the dependencies above. Those that remain pinned are:

• pandas
• scipy
• shap