Cannot Initialize Ray

[yangky11]

What happened + What you expected to happen

Hi,

I got the following error when calling ray.init(). The code worked before, and it is likely related to pydantic releasing a new version (2.0) today.

2023-06-30 23:34:20,508	ERROR services.py:1207 -- Failed to start the dashboard , return code 1
2023-06-30 23:34:20,508	ERROR services.py:1232 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2023-06-30 23:34:20,532	ERROR services.py:1276 -- 
The last 20 lines of /tmp/ray/session_2023-06-30_23-34-17_565773_79053/logs/dashboard.log (it contains the error message from the dashboard): 
  File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/ray/dashboard/modules/log/log_manager.py", line 8, in <module>
    from ray.util.state.common import (
  File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/ray/util/state/__init__.py", line 1, in <module>
    from ray.util.state.api import (
  File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/ray/util/state/api.py", line 17, in <module>
    from ray.util.state.common import (
  File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/ray/util/state/common.py", line 120, in <module>
    @dataclass(init=True)
  File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/pydantic/dataclasses.py", line 139, in dataclass
    assert init is False, 'pydantic.dataclasses.dataclass only supports init=False'
AssertionError: pydantic.dataclasses.dataclass only supports init=False
2023-06-30 23:34:21,181	INFO worker.py:1636 -- Started a local Ray instance.
[2023-06-30 23:34:22,874 E 79053 5475804] core_worker.cc:193: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

Versions / Dependencies

# packages in environment at /Users/kaiyuy/miniconda3/envs/lean-dojo:
#
# Name                    Version                   Build  Channel
aiohttp                   3.8.4                    pypi_0    pypi
aiohttp-cors              0.7.0                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
annotated-types           0.5.0                    pypi_0    pypi
async-timeout             4.0.2                    pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
blessed                   1.20.0                   pypi_0    pypi
ca-certificates           2023.05.30           hecd8cb5_0  
cachetools                5.3.1                    pypi_0    pypi
certifi                   2023.5.7                 pypi_0    pypi
charset-normalizer        3.1.0                    pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
colorful                  0.5.5                    pypi_0    pypi
distlib                   0.3.6                    pypi_0    pypi
filelock                  3.12.2                   pypi_0    pypi
frozenlist                1.3.3                    pypi_0    pypi
google-api-core           2.11.1                   pypi_0    pypi
google-auth               2.21.0                   pypi_0    pypi
googleapis-common-protos  1.59.1                   pypi_0    pypi
gpustat                   1.1                      pypi_0    pypi
grpcio                    1.49.1                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
jsonschema                4.17.3                   pypi_0    pypi
libcxx                    14.0.6               h9765a3e_0  
libffi                    3.4.4                hecd8cb5_0  
msgpack                   1.0.5                    pypi_0    pypi
multidict                 6.0.4                    pypi_0    pypi
ncurses                   6.4                  hcec6c5f_0  
numpy                     1.25.0                   pypi_0    pypi
nvidia-ml-py              11.525.131               pypi_0    pypi
opencensus                0.11.2                   pypi_0    pypi
opencensus-context        0.1.3                    pypi_0    pypi
openssl                   3.0.9                hca72f7f_0  
packaging                 23.1                     pypi_0    pypi
pip                       23.1.2           py39hecd8cb5_0  
platformdirs              3.8.0                    pypi_0    pypi
prometheus-client         0.17.0                   pypi_0    pypi
protobuf                  4.23.3                   pypi_0    pypi
psutil                    5.9.5                    pypi_0    pypi
py-spy                    0.3.14                   pypi_0    pypi
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pydantic                  2.0                      pypi_0    pypi
pydantic-core             2.0.1                    pypi_0    pypi
pyrsistent                0.19.3                   pypi_0    pypi
python                    3.9.16               h5ee71fb_3  
ray                       2.5.1                    pypi_0    pypi
readline                  8.2                  hca72f7f_0  
requests                  2.31.0                   pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
setuptools                67.8.0           py39hecd8cb5_0  
six                       1.16.0                   pypi_0    pypi
smart-open                6.3.0                    pypi_0    pypi
sqlite                    3.41.2               h6c40b1e_0  
tk                        8.6.12               h5d9f67b_0  
typing-extensions         4.7.0                    pypi_0    pypi
tzdata                    2023c                h04d1e81_0  
urllib3                   1.26.16                  pypi_0    pypi
virtualenv                20.21.0                  pypi_0    pypi
wcwidth                   0.2.6                    pypi_0    pypi
wheel                     0.38.4           py39hecd8cb5_0  
xz                        5.4.2                h6c40b1e_0  
yarl                      1.9.2                    pypi_0    pypi
zlib                      1.2.13               h4dc903c_0  

Reproduction script

import ray
ray.init()

Issue Severity

High: It blocks me from completing my task.

[pcmoritz]

Thanks for the report! This is indeed very likely related to the pydantic 2 release, see #36990

As a workaround, try to run pip install "pydantic<2" -- does that work for you?

We already have a fix #37000 that will be released in Ray 2.6

[yangky11]

@pcmoritz Thanks! Do you happen to have any estimate of when Ray 2.6 will be released?

[alanwguo]

There is some work in progress to allow ray to support both pydantic 1.x and 2.x concurrently.
Unfortunately, ray[serve] depends on fast-api which has not yet been updated to support pydantic 2.x

#37055

[jaidisido]

Folks any timeline on a permanent fix here please? A significant share of our users are reaching out after being impacted by this issue

[rkooo567]

@jaidisido the PR is pretty close to be merged, and is highly likely to be included in 2.6 release. #37055

[gilvikra]

+1. All cluster creations are failing

[scv119]

For everyone affected by this issue, can you try to use pydantic 1.x? pip install "pydantic<2"?

[jaidisido]

@scv119 we are already aware of that workaround and I can confirm that it works. However a permanent fix is desirable here as the error is too cryptic and too many users have to look around/raise an issue before discovering the fix

[PhilippWillms]

For everyone affected by this issue, can you try to use pydantic 1.x? pip install "pydantic<2"?

Yes

[alanwguo]

We've already pinned pydantic to be less than 2 and that will be released with ray 2.6. After that fix, folks that install the latest version of Ray will always install a compatible version of pydantic and the error will not appear.

To have ray support both pydantic 1.x and 2.x at the same time is more difficult and also depends on other libraries adding support for pydantic 2.x. For example, ray serve depends on fastapi (one of the most popular python libraries), and fastapi is not yet compatible with pydantic 2.x.

[pdames]

We're also impacted by this issue on the Iceberg branch of DeltaCAT now that Iceberg merged apache/iceberg#7782.

[Fokko]

@pdames Thanks for raising this, do you know what the timeline for Ray is to move to Pydantic 2?

[martalist]

fastapi is not yet compatible with pydantic 2.x

@alanwguo it's now supported, as of FastAPI v0.100.0. Pydantic 2 seems to have significant performance improvements. Do you know if Ray Serve will be updated anytime soon?

[edoakes]

@martalist we've run into a number of issues related to serialization in order to support Pydantic 2 in Ray Serve. We are currently targeting the 2.8 release to address these.

[ddelange]

ref #40451

[shrekris-anyscale]

We've merged the changes to make Ray compatible with Pydantic 2.5+. You can start using Pydantic 2.5+ with Ray 2.9, which should be out at the end of December.

These changes should also be in the Ray nightly, so feel free to try them out!