What dask pins does qhub require?

[kcpevey]

I had been using the following pins for my qhub envs:

  - dask-gateway==0.6.1
  - dask==2.14.0
  - distributed==2.14.0
  - python-blosc==1.9.1

Is this still the required set of pins?

[rsignell-usgs]

@kcpevey, I think questions here on discussions unfortunately don't show up on the radar screens of quansight folks.

I think it's good to use discussion for these sorts of questions, but then perhaps we should post the discussion link on the Qhub Gitter to get attention.

[laisbsc]

Is this still the required set of pins?
@aktech @tylerpotts do we know the answer to Kim's question?

[aktech]

Yup, see the note below Environments here:
https://github.com/Quansight/qhub-cloud/blob/main/docs/docs/gcp/configuration.md#environments

[kcpevey]

Hey @rsignell-usgs! Thanks for letting me know and for bumping the question so I could get an answer! I'm working with @dharhas at Quansight these days. I'll raise the issue of keeping up with this board - I think it's important for the community :)

[tylerpotts]

@rsignell-usgs The best way to manage dask pins now is to use the qhub-dask metapackage from conda-forge. Usage will look like this:

environments:
  environment-dask.yaml:
    name: dask
    channels:
    - conda-forge
    dependencies:
    - python
    - ipykernel
    - ipywidgets
    - qhub-dask ==0.2.3

The pins for the metapackage can be found in the conda-forge recipe here: https://github.com/conda-forge/qhub-dask-meta-feedstock/blob/master/recipe/meta.yaml

[brl0]

To my knowledge, since qhub >= 0.3.* is on the most recent version of dask gateway (v0.9.0), it probably no longer requires pinning older versions of dask and distributed, and I think that to date, the latest versions of dask and distributed should work as well.

It looks like the pins in the qhub-dask-meta == 0.2.3 are still pointing to the versions of dask needed for qhub <= 0.3. We ran into issues with the versions pinned here on the latest version of qhub, and to my knowledge they don't work.

Based on the dask gateway dockerfile, they are using dask 2.3.0 and distributed 2.3.1, which I would probably consider as the minimum bound of the versions of dask to use with the current version of dask gateway.

Also note this comment from @costrouc:

Also we need qhub-dask to follow the qhub version with the pinning requirements. e.g. version 0.2.3 of qhub will require qhub-dask 0.2.3

So it appears that qhub-dask-meta has not yet been updated for the 0.3.* series.

It would be great if the pins in the qhub-dask-meta package were being updated regularly based on what is known to work at the time. That said, I can appreciate that this would probably benefit from qhub including tests for dask gateway functionality, which I don't think has been added yet, but may be planned or intended at some point on the roadmap.

In general, I have concerns about having so many version pins without some mechanisms for keeping them reasonably up to date. To be fair, it is possible that my perspective has become a bit biased after some difficult dask issues I have dealt with while stuck on an old version. I can also appreciate the need for stability in managing the complexity of such a complicated stack of technology, as indicated by @costrouc here:

I think we should favor pinning for stability

I wonder if, to help address this concern, it might help if there was something like a checklist of all the tools, dependencies, images, charts, providers, etc that are pinned along with links to those where those pins are in the code, and possibly even with associated links to repos and registries. My thinking here is it might make it easier for users and new contributors to try updating things to allow for more ad-hoc or haphazard testing.

And finally, for yet another shameless plug, I think such ad-hoc or haphazard testing and integration of updated versions of all the various components would be greatly improved and simplified by incorporating this suggestion.

Sorry for the long winded rant discussion. :)

[rsignell-usgs]

I understand pinning dask-gateway in the qhub-dask metapackage, but why dask distributed?
From: https://github.com/conda-forge/qhub-dask-feedstock/blob/main/recipe/meta.yaml#L14-L15

requirements:
  run:
    - dask
    - distributed ==2022.04.2
    - dask-gateway ==2022.4.0
    - click ==8.0.2

which results in my ESIP qhub/nebari environment having:

13:39 $ conda list dask
# packages in environment at /home/conda/users/envs/pangeo:
#
# Name                    Version                   Build  Channel
dask                      2022.4.2           pyhd8ed1ab_0    conda-forge
dask-core                 2022.4.2           pyhd8ed1ab_0    conda-forge
dask-gateway              2022.4.0           pyh8af1aa0_0    conda-forge
dask-geopandas            0.2.0              pyhd8ed1ab_0    conda-forge
dask-image                2022.9.0           pyhd8ed1ab_0    conda-forge
qhub-dask                 0.4.4                hd8ed1ab_0    conda-forge

I'd like to use the new memory management in 2022.11.0 on our ESIP qhub.

[alimanfoo]

Seems like @rsignell-usgs you and I are cosmically synchronised :) I came here to ask pretty much the same thing, specifically...

• I would like to use dask >= 2022.11.0 in order to take advantage of the new queueing scheduler
• I would like to use distributed >= 2022.9.2 in order to take advantage of active memory management

Both of these I think are going to be pretty game-changing for being able to run computations that previously wouldn't run at all.

I'd really like to know whether we need to strictly stick to the pinning in the nebari-dask (formerly qhub-dask) metapackage or not?

If so, what is the schedule for upgrading to these newer versions of dask and distributed?

(And why is distributed getting pinned?)

[rsignell-usgs]

@alimanfoo good to know another qhub/nebari user1 (besides the folks on the ESIP deployment) 😄
I'm trying an environment build right now without the qhub-dask metapackage and with just:

- dask-gateway==2022.4.0
- click==8.0.2

I think it should work. I'll let you know!

[alimanfoo]

Thanks @rsignell-usgs 😎 Will be great to know if that works. Out of interest what version of qhub/nebari are you using? We're about to create a new deployment of qhub/nebari and I'd really like to know if we can go ahead with an existing version or whether we'd need to wait to be able to use latest dask/distributed.

[rsignell-usgs]

@alimanfoo , as hoped, pinning dask gateway and click but unpinning dask worked fine. Here's the ESIP qhub config: https://gist.github.com/rsignell-usgs/0eb26dadc6a0e0c3f8a9c0e6bb06a4d2

If you take a look at the config, you can see we modified the pdash environment specification, and that environment now has:

06:40 $ conda activate pdash
(pdash) rsignell-usgs:~ 
06:40 $ conda list dask
# packages in environment at /home/conda/filesystem/envs/pdash:
#
# Name                    Version                   Build  Channel
dask                      2022.11.0          pyhd8ed1ab_0    conda-forge
dask-core                 2022.11.0          pyhd8ed1ab_0    conda-forge
dask-gateway              2022.4.0           pyh8af1aa0_0    conda-forge
dask-geopandas            0.2.0              pyhd8ed1ab_0    conda-forge
dask-image                2022.9.0           pyhd8ed1ab_0    conda-forge

And it works great! I ran your original test case (I used the notebook here) with both dask 2022.4.0 and with 2022.11.0:

2022.4.0:

2022.11.0:

[dharhas]

Chiming in here. The original reason for pinning all three was that in the early days (i.e. 2 years ago) we had a lot of instability when dask-gateway and distributed did not match. If we are confident that only dask-gateway needs to be pinned we could change nebari-dask metapackage to only pin gateway.

But we wanted our dashboard apps to see the new dask as well, and the dashboard apps only see the environments defined in the config. At least with v0.4.4 <- This is fixed in the latest release

[dharhas]

or we could add an overide in conda-store that always inserts the correct dask-gateway if dask is in the environment... @costrouc can we do this easily?

[rsignell-usgs]

@dharhas I don't know kind of testing should be done, but the notebooks I've tested so far using dask gateway clusters with the latest version of dask and distributed work fine.

[alimanfoo]

Thanks @rsignell-usgs and @dharhas for the additional info, very useful.

[iameskild]

FWIW the nebari-dask==2022.11.1 meta package should be using the most recent versions of dask/dask-gateway/distributed.