[ci] prevent getting incompatible dask and distributed versions

[jameslamb]

Python CI jobs in several pull requests are failing right now. For example, see #4048 (comment) and https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=9396&view=logs&j=c28dceab-947a-5848-c21f-eef3695e5f11&t=fa158246-17e2-53d4-5936-86070edbaacf .

All of the Dask tests are failing with an error like this

E AttributeError: 'HighLevelGraph' object has no attribute 'dask_distributed_pack'

This error occurs when you try to use dask 2021.2.0 with distributed 2021.3.0. For more documentation on that, see dask/dask#7331.

Specifically, LightGBM's CI is hitting this error because it installs dask and distributed from the default anaconda channel, which is updated more slowly than conda-forge.

I see the following in the logs from one of the failed jobs.

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ...
    dask-2021.2.0              |     pyhd3eb1b0_0           5 KB
    dask-core-2021.2.0         |     pyhd3eb1b0_0         643 KB
    ...
    distributed-2021.3.0       |   py39h06a4308_0         1.0 MB

This PR proposes installing dask and distributed from conda-forge, and putting an explicit version floor of 2021.3.0 on those libraries.

[StrikerRUS]

Thanks for the prompt fix!
I'm OK with these changes as they are really needed to unblock our CI. However, I'd like to remind that mixing installations from different conda channels can be very messy and always leads to significant increasing of installation time and consumed memory.
ContinuumIO/anaconda-issues#11604 (comment)

[StrikerRUS]

So we should revert migration to conda-forge right after the needed versions will be available on default conda channel.

[jameslamb]

So we should revert migration to conda-forge right after the needed versions will be available on default conda channel.

ok

[jameslamb]

several Azure jobs failed with what looks like infrastructure problems

##[error]We stopped hearing from agent lgbubb5e00004AJ. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink/?linkid=846610
Agent: lgbubb5e00004AJ
Started: Today at 9:20 AM
Duration: 51m 51s

https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=9406&view=logs&j=5ea309bb-dea2-5bcb-538b-5c76ecf159eb

Rebuilding by pushing an empty commit, so the logs are still there to go back and look at.

[jameslamb]

I'm not sure what's happening, but I see 2 check-docs tasks. One is marked failed and one marked successful.

Weirdly, that failing one isn't blocking merge and it didn't cause Static Analysis / all-successful to fail. So maybe it is just a "ghost job" created accidentally by GitHub?

Since there was a successful one and since this PR is blocking other PRs, I'm going to merge this. But it's still very strange.

• failed check: https://github.com/microsoft/LightGBM/pull/4054/checks?check_run_id=2068433804
• successful check: https://github.com/microsoft/LightGBM/pull/4054/checks?check_run_id=2068835237

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.