Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dask setting to disable query-planning for now #6149

Merged
merged 8 commits into from
Mar 14, 2024
Merged

Conversation

hoxbro
Copy link
Member

@hoxbro hoxbro commented Mar 13, 2024

Our CI test is broken with the latest Dask release. This restores the previous behavior.

@hoxbro hoxbro added the type: compatibility Compability with upstream packages label Mar 13, 2024
holoviews/tests/conftest.py Outdated Show resolved Hide resolved
@codecov-commenter
Copy link

codecov-commenter commented Mar 13, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 23.79%. Comparing base (075ae6e) to head (a755098).

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #6149       +/-   ##
===========================================
- Coverage   88.69%   23.79%   -64.90%     
===========================================
  Files         316      316               
  Lines       66017    66014        -3     
===========================================
- Hits        58554    15711    -42843     
- Misses       7463    50303    +42840     
Flag Coverage Δ
ui-tests 23.79% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hoxbro
Copy link
Member Author

hoxbro commented Mar 13, 2024

I can see the unit tests are not finishing up with this being the last logs:

==== 4903 passed, 90 skipped, 8 deselected, 1 xfailed in 649.88s (0:10:49) =====
Unable to join threads to shut down before fork(). This can break multithreading in child processes.
Unable to join threads to shut down before fork(). This can break multithreading in child processes.
Unable to join threads to shut down before fork(). This can break multithreading in child processes.
Unable to join threads to shut down before fork(). This can break multithreading in child processes.

setup.py Outdated Show resolved Hide resolved
@hoxbro
Copy link
Member Author

hoxbro commented Mar 13, 2024

Diff between last good and bad (holoviz artifact holoviews 2267 2273)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
renamed: home/shh/.cache/holoviz-cli/artifact/holoviews_2273/environment_unit_test_suite_3.11_Linux.yml to home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:40 @ dependencies:
   - bleach=6.1.0=pyhd8ed1ab_0
   - blinker=1.7.0=pyhd8ed1ab_0
   - blosc=1.21.5=h0f2a231_0
-  - bokeh=3.4.0rc2=py_0
+  - bokeh=3.4.0.dev8=py_0
   - boltons=23.1.1=pyhd8ed1ab_0
   - brotli=1.1.0=hd590300_1
   - brotli-bin=1.1.0=hd590300_1
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:64 @ dependencies:
   - codecov=2.1.13=pyhd8ed1ab_0
   - colorama=0.4.6=pyhd8ed1ab_0
   - colorcet=3.1.0=py_0
-  - comm=0.2.2=pyhd8ed1ab_0
+  - comm=0.2.1=pyhd8ed1ab_0
   - conda=24.1.2=py311h38be061_0
   - conda-build=24.1.2=py311h38be061_0
   - conda-index=0.4.0=pyhd8ed1ab_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:74 @ dependencies:
   - contourpy=1.2.0=py311h9547e67_0
   - coverage=7.4.3=py311h459d7ec_1
   - cycler=0.12.1=pyhd8ed1ab_0
+  - cytoolz=0.12.3=py311h459d7ec_0
   - dash=2.16.1=pyhd8ed1ab_0
-  - dask-core=2023.2.1=pyhd8ed1ab_0
+  - dask=2024.2.1=pyhd8ed1ab_0
+  - dask-core=2024.2.1=pyhd8ed1ab_0
   - datashader=0.16.1a1=py_0
   - dav1d=1.2.1=hd590300_0
   - debugpy=1.8.1=py311hb755f60_0
   - decorator=5.1.1=pyhd8ed1ab_0
   - defusedxml=0.7.1=pyhd8ed1ab_0
+  - distributed=2024.2.1=pyhd8ed1ab_0
   - distro=1.9.0=pyhd8ed1ab_0
   - doit=0.36.0=pyhd8ed1ab_0
   - entrypoints=0.4=pyhd8ed1ab_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:137 @ dependencies:
   - imagecodecs=2024.1.1=py311hd0e15ba_2
   - imageio=2.34.0=pyh4b66e23_0
   - importlib-metadata=7.0.2=pyha770c72_0
-  - importlib-resources=6.3.0=pyhd8ed1ab_0
+  - importlib-resources=6.1.3=pyhd8ed1ab_0
   - importlib_metadata=7.0.2=hd8ed1ab_0
-  - importlib_resources=6.3.0=pyhd8ed1ab_0
+  - importlib_resources=6.1.3=pyhd8ed1ab_0
   - iniconfig=2.0.0=pyhd8ed1ab_0
   - ipykernel=6.29.3=pyhd33586a_0
   - ipython=8.22.2=pyh707e725_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:157 @ dependencies:
   - jupyter-lsp=2.2.4=pyhd8ed1ab_0
   - jupyter_client=8.6.1=pyhd8ed1ab_0
   - jupyter_core=5.7.2=py311h38be061_0
-  - jupyter_events=0.9.1=pyhd8ed1ab_0
+  - jupyter_events=0.9.0=pyhd8ed1ab_0
   - jupyter_server=2.13.0=pyhd8ed1ab_0
-  - jupyter_server_terminals=0.5.3=pyhd8ed1ab_0
+  - jupyter_server_terminals=0.5.2=pyhd8ed1ab_0
   - jupyterlab=4.1.4=pyhd8ed1ab_0
   - jupyterlab_pygments=0.3.0=pyhd8ed1ab_1
   - jupyterlab_server=2.25.4=pyhd8ed1ab_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:175 @ dependencies:
   - libabseil=20240116.1=cxx17_h59595ed_2
   - libaec=1.1.2=h59595ed_1
   - libarchive=3.7.2=h2aa1ff5_1
-  - libarrow=15.0.1=he513d67_1_cpu
-  - libarrow-acero=15.0.1=h59595ed_1_cpu
-  - libarrow-dataset=15.0.1=h59595ed_1_cpu
-  - libarrow-flight=15.0.1=hc6145d9_1_cpu
-  - libarrow-flight-sql=15.0.1=h757c851_1_cpu
-  - libarrow-gandiva=15.0.1=hb016d2e_1_cpu
-  - libarrow-substrait=15.0.1=h757c851_1_cpu
+  - libarrow=15.0.1=h5476b98_0_cpu
+  - libarrow-acero=15.0.1=h59595ed_0_cpu
+  - libarrow-dataset=15.0.1=h59595ed_0_cpu
+  - libarrow-flight=15.0.1=hc6145d9_0_cpu
+  - libarrow-flight-sql=15.0.1=h757c851_0_cpu
+  - libarrow-gandiva=15.0.1=hb016d2e_0_cpu
+  - libarrow-substrait=15.0.1=h757c851_0_cpu
   - libass=0.17.1=h8fe9dca_1
   - libavif16=1.0.4=h1dcd450_1
   - libblas=3.9.0=21_linux64_openblas
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:238 @ dependencies:
   - libopenvino-tensorflow-frontend=2024.0.0=hca94c1a_1
   - libopenvino-tensorflow-lite-frontend=2024.0.0=h59595ed_1
   - libopus=1.3.1=h7f98852_1
-  - libparquet=15.0.1=h352af49_1_cpu
+  - libparquet=15.0.1=h352af49_0_cpu
   - libpciaccess=0.18=hd590300_0
   - libpng=1.6.43=h2797004_0
   - libprotobuf=4.25.3=h08a7969_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:268 @ dependencies:
   - linkify-it-py=2.0.3=pyhd8ed1ab_0
   - llvmlite=0.42.0=py311ha6695c7_1
   - locket=1.0.0=pyhd8ed1ab_0
+  - lz4=4.3.3=py311h38e4bf4_0
   - lz4-c=1.9.4=hcb278e6_0
   - lzo=2.10=h516909a_1000
   - markdown=3.5.2=pyhd8ed1ab_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:281 @ dependencies:
   - menuinst=2.0.2=py311h38be061_0
   - mistune=3.0.2=pyhd8ed1ab_0
   - more-itertools=10.2.0=pyhd8ed1ab_0
+  - msgpack-python=1.0.7=py311h9547e67_0
   - multipledispatch=0.6.0=py_0
   - munkres=1.1.4=pyh9f0ad1d_0
-  - nbclient=0.10.0=pyhd8ed1ab_0
+  - nbclient=0.8.0=pyhd8ed1ab_0
   - nbconvert=7.16.2=pyhd8ed1ab_0
   - nbconvert-core=7.16.2=pyhd8ed1ab_0
   - nbconvert-pandoc=7.16.2=pyhd8ed1ab_0
-  - nbformat=5.10.2=pyhd8ed1ab_0
+  - nbformat=5.9.2=pyhd8ed1ab_0
   - nbval=0.11.0=pyhd8ed1ab_0
   - ncurses=6.4=h59595ed_2
   - nest-asyncio=1.6.0=pyhd8ed1ab_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:304 @ dependencies:
   - openh264=2.4.1=h59595ed_0
   - openjpeg=2.5.2=h488ebb8_0
   - openssl=3.2.1=hd590300_0
-  - orc=2.0.0=h1e5e2c1_0
+  - orc=1.9.2=h1e5e2c1_3
   - outcome=1.3.0.post0=pyhd8ed1ab_0
   - overrides=7.7.0=pyhd8ed1ab_0
   - p11-kit=0.24.1=hc5aa10d_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:312 @ dependencies:
   - pandas=2.2.1=py311h320fe9a_0
   - pandoc=3.1.12.2=ha770c72_0
   - pandocfilters=1.5.0=pyhd8ed1ab_0
-  - panel=1.4.0b3=py_0
+  - panel=1.4.0b2=py_0
   - pango=1.52.1=ha41ecd1_0
-  - param=2.1.0rc1=py_0
+  - param=2.1.0a2=py_0
   - parso=0.8.3=pyhd8ed1ab_0
   - parsy=2.1=pyhd8ed1ab_0
   - partd=1.4.1=pyhd8ed1ab_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:342 @ dependencies:
   - pure_eval=0.2.2=pyhd8ed1ab_0
   - py=1.11.0=pyh6c4a22f_0
   - py-lief=0.12.3=py311ha362b79_0
-  - pyarrow=15.0.1=py311h39c9aba_1_cpu
+  - pyarrow=15.0.1=py311h39c9aba_0_cpu
   - pyarrow-hotfix=0.6=pyhd8ed1ab_0
   - pybind11-abi=4=hd8ed1ab_3
   - pycosat=0.6.6=py311h459d7ec_0
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:395 @ dependencies:
   - selenium=4.18.1=pyhd8ed1ab_0
   - selenium-manager=4.18.1=he8a937b_0
   - send2trash=1.8.2=pyh41d4057_0
-  - setuptools=69.2.0=pyhd8ed1ab_0
+  - setuptools=69.1.1=pyhd8ed1ab_0
   - shapely=2.0.3=py311h2032efe_0
   - six=1.16.0=pyh6c4a22f_0
   - snappy=1.1.10=h9fff704_0
   - sniffio=1.3.1=pyhd8ed1ab_0
   - sortedcontainers=2.4.0=pyhd8ed1ab_0
   - soupsieve=2.5=pyhd8ed1ab_1
-  - spatialpandas=0.4.10=pyhd8ed1ab_1
+  - spatialpandas=0.4.10=py_0
   - sqlalchemy=2.0.28=py311h459d7ec_0
   - sqlglot=20.11.0=pyhd8ed1ab_0
   - stack_data=0.6.2=pyhd8ed1ab_0
   - streamz=0.6.4=pyh6c4a22f_0
   - svt-av1=1.8.0=h59595ed_0
   - tbb=2021.11.0=h00ab1b0_1
+  - tblib=3.0.0=pyhd8ed1ab_0
   - tenacity=8.2.3=pyhd8ed1ab_0
-  - terminado=0.18.1=pyh0d859eb_0
+  - terminado=0.18.0=pyh0d859eb_0
   - tifffile=2024.2.12=pyhd8ed1ab_0
   - tinycss2=1.2.1=pyhd8ed1ab_0
   - tk=8.6.13=noxft_h4845f30_101
@ home/shh/.cache/holoviz-cli/artifact/holoviews_2267/environment_unit_test_suite_3.11_Linux.yml:431 @ dependencies:
   - typing_utils=0.1.0=pyhd8ed1ab_0
   - tzdata=2024a=h0c530f3_0
   - uc-micro-py=1.0.3=pyhd8ed1ab_0
-  - ucx=1.15.0=h9929b8b_6
+  - ucx=1.15.0=h9929b8b_4
   - uri-template=1.3.0=pyhd8ed1ab_0
   - urllib3=2.2.1=pyhd8ed1ab_0
   - wcwidth=0.2.13=pyhd8ed1ab_0

@hoxbro
Copy link
Member Author

hoxbro commented Mar 13, 2024

It is the latest panel dev release, which caused the unit test never to finish.

A git bisect with this command: timeout 30 pytest ../holoviews/holoviews/tests/plotting/bokeh/test_server.py shows that holoviz/panel#6459 causes it

@hoxbro
Copy link
Member Author

hoxbro commented Mar 13, 2024

@philippjfr, can you take a look at test_server.py? And see if there is an easy way to fix this.

@hoxbro hoxbro marked this pull request as draft March 13, 2024 20:02
@philippjfr
Copy link
Member

😭I spent like 3 hours on this last night.

setup.py Outdated Show resolved Hide resolved
[skip ci]
@philippjfr
Copy link
Member

philippjfr commented Mar 14, 2024

I would suggest using the serve_and_wait and serve_and_request utilities in panel.tests.util to manage the servers. It seems not going via those utilities resulted in a zombie thread.

@philippjfr
Copy link
Member

Here's the fix in panel holoviz/panel#6492

.github/workflows/test.yaml Outdated Show resolved Hide resolved
@hoxbro hoxbro marked this pull request as ready for review March 14, 2024 09:34
@hoxbro hoxbro merged commit 3b4a214 into main Mar 14, 2024
5 of 11 checks passed
@hoxbro hoxbro deleted the setting_dask branch March 14, 2024 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: compatibility Compability with upstream packages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants