[ci] python-package "regular" CI jobs failing to install dependencies

[jameslamb]

Description

The TASK=regular jobs in this project's CI run the Python package's unit tests and test a few other things:

• the example notebookes in the examples/ directory
• that lightgbm can be imported after all of its dependencies have been removed

Those jobs are currently failing across all operating systems, with an error like this:

[NbConvertApp] Converting notebook interactive_plot_example.ipynb to notebook
[IPKernelApp] ERROR | KeyboardInterrupt caught in kernel.
[NbConvertApp] Writing 178679 bytes to interactive_plot_example.ipynb

PackagesNotFoundError: The following packages are missing from the target environment:
  - dask

Error: Process completed with exit code 255.

Reproducible example

Logs from latest commit on master:

• Azure DevOps Linux_latest regular (link)
• Azure DevOps macOS regular (link)
• GitHub Actions regular (macOS-latest, Python 3.9) (link)

Environment info

LightGBM version or commit hash: 255c93b

Additional Comments

I don't see any of the dependencies like dask, dask-core, etc. getting installed in the logs.

It seems that this might be silently failing:

LightGBM/.ci/test.sh

Lines 128 to 135 in 255c93b

	CONSTRAINED_DEPENDENCIES="'dask-core>=2023.5.0' 'distributed>=2023.5.0' 'pandas>=2.0'"
	if [[ $PYTHON_VERSION == "3.7" ]]; then
	CONSTRAINED_DEPENDENCIES="'dask-core' 'distributed' 'pandas<2.0'"
	fi
	# including python=version[build=*cpython] to ensure that conda doesn't fall back to pypy
	mamba create -q -y -n $CONDA_ENV \
	${CONSTRAINED_DEPENDENCIES} \

[jmoralez]

I believe the command that's failing is the uninstall, maybe it's due to a difference between installing with mamba and trying to uninstall with conda.

LightGBM/.ci/test.sh

Line 312 in 255c93b

mamba install -q -y -n $CONDA_ENV \

LightGBM/.ci/test.sh

Line 322 in 255c93b

conda uninstall --force --yes \

I can submit a PR with that change if you think that may be it.

[jameslamb]

ooooo interesting. Yeah I'd welcome that PR, thank you!

[jmoralez]

Oh sorry I hadn't seen #6283, do you want me to create a new one or push to that branch?

[jameslamb]

whatever's easier for you! No preference.

[jameslamb]

If that doesn't work, could you test another idea I just had?

Maybe the problem is that the package we install is named dask-core

LightGBM/.ci/test.sh

Line 128 in 255c93b

CONSTRAINED_DEPENDENCIES="'dask-core>=2023.5.0' 'distributed>=2023.5.0' 'pandas>=2.0'"

but then we're asking conda uninstall to remove a package called dask.

LightGBM/.ci/test.sh

Lines 322 to 325 in 255c93b

	conda uninstall --force --yes \
	cffi \
	dask \
	distributed \

Maybe something has changed recently in conda, mamba, or the dask-core metapackage that breaks that.

Either way, I think we should probably be adding dask-core to that conda uninstall call.