Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dashboard not loading with Cythonized Scheduler #4760

Closed
jakirkham opened this issue Apr 28, 2021 · 8 comments
Closed

Dashboard not loading with Cythonized Scheduler #4760

jakirkham opened this issue Apr 28, 2021 · 8 comments

Comments

@jakirkham
Copy link
Member

This appears to have started recently. Possibly related to PR ( #4651 ) ( cc @crusaderky )? In particular loading the dashboard shows a 500 error. Locally the following exception comes up when the Scheduler is Cythonized. The issue does not occur when pure Python is used.

cc @quasiben

What happened:

Screen Shot 2021-04-27 at 5 24 17 PM

distributed.utils - ERROR - object.__new__(distributed.scheduler.MemoryState) is not safe, use distributed.scheduler.MemoryState.__new__()
Traceback (most recent call last):
  File "/Users/jkirkham/Developer/distributed/distributed/utils.py", line 668, in log_errors
    yield
  File "/Users/jkirkham/Developer/distributed/distributed/dashboard/components/scheduler.py", line 296, in update
    meminfo = self.scheduler.memory
  File "distributed/scheduler.py", line 1913, in distributed.scheduler.SchedulerState.memory.__get__
  File "distributed/scheduler.py", line 330, in distributed.scheduler.MemoryState.sum
TypeError: object.__new__(distributed.scheduler.MemoryState) is not safe, use distributed.scheduler.MemoryState.__new__()

What you expected to happen:

Minimal Complete Verifiable Example:

import webbrowser
from dask.distributed import Client

client = Client()

webbrowser.open(client.cluster.dashboard_link)

Anything else we need to know?:

Environment:

  • Dask version: 2021.04.1
  • Python version: 3.8.8
  • Operating System: macOS
  • Install method (conda, pip, source): Conda
@jrbourbeau
Copy link
Member

Thanks for surfacing this @jakirkham! I tried to reproduce locally but was able to see the dashboard as usual (no errors were logged). I first checked out the latest distributed commit locally (b577ece) and then, to cythonize the scheduler, followed these instructions. Is that all I need to do locally, or is there another step?

I've also included the conda list output for my local environment in case that's useful (note I'm using the latest cython=0.29.23 release on conda-forge)

Packages I have installed locally (conda list): 
# packages in environment at /Users/james/miniforge3/envs/distributed:
#
# Name                    Version                   Build  Channel
aiobotocore               1.2.2              pyhd8ed1ab_0    conda-forge
aiohttp                   3.7.4            py39h89e85a6_0    conda-forge
aioitertools              0.7.1              pyhd8ed1ab_0    conda-forge
alabaster                 0.7.12                   pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
appnope                   0.1.2            py39h6e9494a_1    conda-forge
argon2-cffi               20.1.0           py39hcbf5805_2    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
async_generator           1.10                       py_0    conda-forge
asyncssh                  2.5.0              pyhd8ed1ab_0    conda-forge
attrs                     20.3.0             pyhd3deb0d_0    conda-forge
babel                     2.9.0                    pypi_0    pypi
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.3              pyhd8ed1ab_0    conda-forge
bcrypt                    3.2.0            py39hcbf5805_1    conda-forge
bleach                    3.3.0              pyh44b312d_0    conda-forge
blosc                     1.21.0               he49afe7_0    conda-forge
bokeh                     2.3.0            py39h6e9494a_0    conda-forge
botocore                  1.19.52            pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py39hcbf5805_1001    conda-forge
bzip2                     1.0.8                hc929b4f_4    conda-forge
c-ares                    1.17.1               h0d85af4_1    conda-forge
c-compiler                1.1.3                h35c211d_0    conda-forge
ca-certificates           2020.12.5            h033912b_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cctools                   949.0.1             hd9211c8_21    conda-forge
cctools_osx-64            949.0.1             h6407bdd_21    conda-forge
certifi                   2020.12.5        py39h6e9494a_1    conda-forge
cffi                      1.14.5           py39h319c39b_0    conda-forge
cfgv                      3.2.0                    pypi_0    pypi
cftime                    1.4.1            py39h329c335_0    conda-forge
chardet                   4.0.0            py39h6e9494a_1    conda-forge
clang                     11.1.0               h694c41f_0    conda-forge
clang-11                  11.1.0          default_he082bbe_0    conda-forge
clang_osx-64              11.1.0               hb91bd55_1    conda-forge
clangxx                   11.1.0          default_he082bbe_0    conda-forge
clangxx_osx-64            11.1.0               h7e1b574_1    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
cloudpickle               1.6.0                      py_0    conda-forge
colorama                  0.4.4                    pypi_0    pypi
compiler-rt               11.1.0               h654b07c_0    conda-forge
compiler-rt_osx-64        11.1.0               h8c5fa43_0    conda-forge
compilers                 1.1.3                h694c41f_0    conda-forge
cryptography              3.4.7            py39ha2c9959_0    conda-forge
curl                      7.76.0               h06286d4_0    conda-forge
cxx-compiler              1.1.3                h926bf3e_0    conda-forge
cython                    0.29.23          py39h9fcab8e_0    conda-forge
cytoolz                   0.11.0           py39hcbf5805_3    conda-forge
dask                      2021.4.1                  dev_0    <develop>
dask-sphinx-theme         1.3.5                    pypi_0    pypi
decorator                 5.0.5              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distlib                   0.3.1                    pypi_0    pypi
distributed               2021.4.1+2.gb577ece3          pypi_0    pypi
docutils                  0.16                     pypi_0    pypi
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
filelock                  3.0.12                   pypi_0    pypi
filesystem-spec           0.2.0                      py_0    conda-forge
fortran-compiler          1.1.3                h6818c3e_0    conda-forge
freetype                  2.10.4               h4cff582_1    conda-forge
fsspec                    0.9.0+2.gbb7c233          pypi_0    pypi
future                    0.18.2           py39h6e9494a_3    conda-forge
gfortran_impl_osx-64      9.3.0               h9cc0e5e_22    conda-forge
gfortran_osx-64           9.3.0               h18f7dce_14    conda-forge
gmp                       6.2.1                h2e338ed_0    conda-forge
h5py                      3.1.0           nompi_py39hdc2b67d_100    conda-forge
hdf4                      4.2.13            h71d84a9_1004    conda-forge
hdf5                      1.10.6          nompi_hc5d9132_1114    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
identify                  2.2.2                    pypi_0    pypi
idna                      2.10               pyh9f0ad1d_0    conda-forge
imagesize                 1.2.0                    pypi_0    pypi
importlib-metadata        3.10.0           py39h6e9494a_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ipykernel                 5.5.3            py39h71a6800_0    conda-forge
ipython                   7.22.0           py39h71a6800_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.6.3              pyhd3deb0d_0    conda-forge
isl                       0.22.1               hb1e8313_2    conda-forge
jedi                      0.18.0           py39h6e9494a_2    conda-forge
jinja2                    2.11.3             pyh44b312d_0    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
joblib                    1.1.0.dev0               pypi_0    pypi
jpeg                      9d                   hbcb3906_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge
jupyter_core              4.7.1            py39h6e9494a_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_widgets        1.0.0              pyhd8ed1ab_1    conda-forge
keras                     2.4.3                    pypi_0    pypi
keyring                   23.0.1                   pypi_0    pypi
krb5                      1.17.2               h60d9502_0    conda-forge
lcms2                     2.12                 h577c468_0    conda-forge
ld64                      530                 hd2e7500_21    conda-forge
ld64_osx-64               530                 he8994da_21    conda-forge
ldid                      2.1.2                h7660a38_2    conda-forge
libblas                   3.9.0                     8_mkl    conda-forge
libcblas                  3.9.0                     8_mkl    conda-forge
libclang-cpp11.1          11.1.0          default_he082bbe_0    conda-forge
libcurl                   7.76.0               h8ef9fac_0    conda-forge
libcxx                    11.1.0               habf9029_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 haf1e3a3_1    conda-forge
libffi                    3.3                  h046ec9c_2    conda-forge
libgfortran               5.0.0           9_3_0_h6c81a4c_21    conda-forge
libgfortran-devel_osx-64  9.3.0               h6c81a4c_22    conda-forge
libgfortran5              9.3.0               h6c81a4c_21    conda-forge
libiconv                  1.16                 haf1e3a3_0    conda-forge
liblapack                 3.9.0                     8_mkl    conda-forge
libllvm11                 11.1.0               hd011deb_2    conda-forge
libnetcdf                 4.7.4           nompi_h9d8a93f_107    conda-forge
libnghttp2                1.43.0               h07e645a_0    conda-forge
libpng                    1.6.37               h7cec526_2    conda-forge
libprotobuf               3.15.7               hcf210ce_0    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libssh2                   1.9.0                h52ee1ee_6    conda-forge
libtiff                   4.2.0                h355d032_0    conda-forge
libwebp-base              1.2.0                h0d85af4_2    conda-forge
llvm-openmp               11.1.0               hda6cdc1_1    conda-forge
llvm-tools                11.1.0               hd011deb_2    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4                       3.1.3            py39heb8eec3_0    conda-forge
lz4-c                     1.9.3                h046ec9c_0    conda-forge
markupsafe                1.1.1            py39hcbf5805_3    conda-forge
mistune                   0.8.4           py39hcbf5805_1003    conda-forge
mkl                       2020.4             h08c4f10_301    conda-forge
more-itertools            8.7.0              pyhd8ed1ab_0    conda-forge
mpc                       1.1.0             ha57cd0f_1009    conda-forge
mpfr                      4.0.2                h72d8aaf_1    conda-forge
msgpack-python            1.0.2            py39hedf5dff_1    conda-forge
multidict                 5.1.0            py39hcbf5805_1    conda-forge
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
nbconvert                 6.0.7            py39h6e9494a_3    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h2e338ed_4    conda-forge
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
netcdf4                   1.5.6           nompi_py39ha7461aa_100    conda-forge
ninja                     1.10.2               h9a9d8cb_0    conda-forge
nodeenv                   1.5.0                    pypi_0    pypi
notebook                  6.3.0            py39h6e9494a_0    conda-forge
numpy                     1.20.2           py39h7eed0ac_0    conda-forge
numpydoc                  1.1.0                    pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openssl                   1.1.1k               h0d85af4_0    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pandas                    1.2.3            py39h4d6be9b_0    conda-forge
pandoc                    2.13                 h0d85af4_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
paramiko                  2.7.2              pyh9f0ad1d_0    conda-forge
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
partd                     1.1.0                      py_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.1.2            py39h5fdd921_0    conda-forge
pip                       21.0.1             pyhd8ed1ab_0    conda-forge
pkginfo                   1.7.0                    pypi_0    pypi
pluggy                    0.13.1           py39h6e9494a_4    conda-forge
pre-commit                2.12.0                   pypi_0    pypi
prometheus_client         0.10.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.18             pyha770c72_0    conda-forge
psutil                    5.8.0            py39hcbf5805_1    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py                        1.10.0             pyhd3deb0d_0    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pygments                  2.8.1              pyhd8ed1ab_0    conda-forge
pynacl                    1.4.0            py39hcbf5805_2    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyrsistent                0.17.3           py39hcbf5805_2    conda-forge
pysocks                   1.7.1            py39h6e9494a_3    conda-forge
pytest                    6.2.3            py39h6e9494a_0    conda-forge
pytest-asyncio            0.12.0           py39hde42818_2    conda-forge
pytest-faulthandler       2.0.1                      py_0    conda-forge
pytest-repeat             0.8.0                      py_0    conda-forge
pytest-rerunfailures      9.1.1                      py_0    conda-forge
pytest-timeout            1.4.2              pyh9f0ad1d_0    conda-forge
python                    3.9.2           h2502468_0_cpython    conda-forge
python-blosc              1.10.2           py39h4d6be9b_0    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-snappy             0.6.0            py39hfa214c3_0    conda-forge
python_abi                3.9                      1_cp39    conda-forge
pytorch                   1.8.0           cpu_py39h4f2e8f6_0    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyyaml                    5.4.1            py39hcbf5805_0    conda-forge
pyzmq                     22.0.3           py39h7fec2f1_1    conda-forge
readline                  8.1                  h05e3726_0    conda-forge
readme-renderer           29.0                     pypi_0    pypi
requests                  2.25.1             pyhd3deb0d_0    conda-forge
requests-toolbelt         0.9.1                    pypi_0    pypi
rfc3986                   1.4.0                    pypi_0    pypi
s3fs                      0.6.0+3.gffe3a52          pypi_0    pypi
scikit-learn              0.24.1           py39h1f17f7f_0    conda-forge
scipy                     1.6.2            py39h056f1c0_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                49.6.0           py39h6e9494a_3    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
sleef                     3.5.1                h35c211d_1    conda-forge
snappy                    1.1.8                hb1e8313_3    conda-forge
snowballstemmer           2.1.0                    pypi_0    pypi
sortedcollections         2.1.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.3.0              pyhd8ed1ab_0    conda-forge
sphinx                    3.5.4                    pypi_0    pypi
sphinx-click              2.7.1                    pypi_0    pypi
sphinx-rtd-theme          0.5.2                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    1.0.3                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.4                    pypi_0    pypi
sqlite                    3.35.4               h44b9ce1_0    conda-forge
tapi                      1100.0.11            h9ce4665_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.9.4            py39h6e9494a_0    conda-forge
testpath                  0.4.4                      py_0    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge
tk                        8.6.10               h0419947_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toolz                     0.11.1                     py_0    conda-forge
torchvision               0.9.0           py39hb9ceeaa_0_cpu    conda-forge
tornado                   6.1              py39hcbf5805_1    conda-forge
tqdm                      4.60.0                   pypi_0    pypi
traitlets                 5.0.5                      py_0    conda-forge
twine                     3.4.1                    pypi_0    pypi
typing-extensions         3.7.4.3                       0    conda-forge
typing_extensions         3.7.4.3                    py_0    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
urllib3                   1.26.4             pyhd8ed1ab_0    conda-forge
virtualenv                20.4.3                   pypi_0    pypi
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
widgetsnbextension        3.5.1            py39h6e9494a_4    conda-forge
wrapt                     1.12.1           py39hcbf5805_3    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
yaml                      0.2.5                haf1e3a3_0    conda-forge
yarl                      1.6.3            py39hcbf5805_1    conda-forge
zeromq                    4.3.4                h1c7c35f_0    conda-forge
zict                      2.0.0                      py_0    conda-forge
zipp                      3.4.1              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h7795811_1010    conda-forge
zstandard                 0.15.2           py39h89e85a6_0    conda-forge
zstd                      1.4.9                h582d3a0_0    conda-forge

@jakirkham
Copy link
Member Author

My guess is it is not using the Cythonized Scheduler then. Don't know offhand why. Maybe there is still some lingering bits from the pure Python Scheduler? Maybe a fresh environment needs to be used?

Ben ran into this when we were doing some other work earlier. Have since reproduced this myself locally as well

@jakirkham jakirkham mentioned this issue Apr 28, 2021
Merged
3 tasks
@jakirkham
Copy link
Member Author

Dug into this a bit locally. The changes in PR ( #4761 ) fix this issue for me. Just to test things a bit further I created an array and computed its sum to make sure something a bit more dynamic worked as well. Switching between tabs seems to work as well

@jakirkham
Copy link
Member Author

As a sanity check to ensure the Cythonized Scheduler is used, it might be worth trying the following

In [1]: import distributed.scheduler

In [2]: distributed.scheduler
Out[2]: <module 'distributed.scheduler' from '/Users/jkirkham/Developer/distributed/distributed/scheduler.cpython-38-darwin.so'>

Note that the filename for the scheduler contains .so instead of .py

@jrbourbeau
Copy link
Member

Alright, I tried in a fresh env and was able to reproduce the 500 error

@crusaderky
Copy link
Collaborator

test_scheduler.py contains extensive unit tests for the MemoryState class. Why don't they trip in CI?

@jrbourbeau jrbourbeau mentioned this issue Apr 28, 2021
Merged
@jakirkham
Copy link
Member Author

Because CI doesn’t test the Cythonized Scheduler correctly

@jakirkham
Copy link
Member Author

Fixed with PR ( #4761 ). Also CI is now covering Cythonization correctly ( #4764 ). Closing this out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jakirkham @crusaderky @jrbourbeau