Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[nit] Make sure Ray Serve installs ray[default] #19262

Closed
triciasfu opened this issue Oct 9, 2021 · 3 comments · Fixed by #19570
Closed

[nit] Make sure Ray Serve installs ray[default] #19262

triciasfu opened this issue Oct 9, 2021 · 3 comments · Fixed by #19570
Assignees
Labels
enhancement Request for new feature and/or capability good first issue Great starter issue for someone just starting to contribute to Ray P1 Issue that should be fixed within a few weeks serve Ray Serve Related Issue

Comments

@triciasfu
Copy link
Contributor

triciasfu commented Oct 9, 2021

No description provided.

@triciasfu triciasfu added enhancement Request for new feature and/or capability serve Ray Serve Related Issue labels Oct 9, 2021
@triciasfu triciasfu added this to the Serve CUJ milestone Oct 9, 2021
@simon-mo simon-mo added the good first issue Great starter issue for someone just starting to contribute to Ray label Oct 11, 2021
@triciasfu triciasfu added the P1 Issue that should be fixed within a few weeks label Oct 11, 2021
@edoakes
Copy link
Contributor

edoakes commented Oct 14, 2021

For this, let's add a test that:

  1. Creates a fresh conda env
  2. Installs ray[serve] only (ideally it'd be from master, maybe not feasible?)
  3. Checks the output of a simple e2e serve script

@simon-mo @architkulkarni could we re-use the runtime_env testing mechanism for installing the wheel built in CI for (2)?

@edoakes
Copy link
Contributor

edoakes commented Oct 14, 2021

I tried running this:

import ray
from ray import serve
serve.start()

@serve.deployment(ray_actor_options={"runtime_env": {"pip": ["requests"]}})
class A:
    pass

A.deploy()

and it hangs without any warning:

(pid=80782) 2021-10-14 09:39:56,527     INFO backend_state.py:896 -- Adding 1 replicas to deployment 'A'. component=serve deployment=A
2021-10-14 09:40:14,882 WARNING worker.py:1227 -- The actor or task with ID ffffffffffffffff0c267d7c776f95260c6091e201000000 cannot be schedule
d right now. You can ignore this message if this Ray cluster is expected to auto-scale or if you specified a runtime_env for this actor or task
, which may take time to install.  Otherwise, this is likely due to all cluster resources being claimed by actors. To resolve the issue, consid
er creating fewer actors or increasing the resources available to this Ray cluster.
Required resources for this actor or task: {CPU_group_7c45bd96450f2713ebbdfbc054264143: 1.000000}
Available resources on this node: {7.000000/8.000000 CPU, 274103400.000000 GiB/274103400.000000 GiB memory, 137051700.000000 GiB/137051700.0000
00 GiB object_store_memory, 1000.000000/1000.000000 bundle_group_7c45bd96450f2713ebbdfbc054264143, 0.000000/1.000000 CPU_group_7c45bd96450f2713ebbdfbc054264143, 1.000000/1.000000 CPU_group_0_7c45bd96450f2713ebbdfbc054264143, 0.980000/1.000000 node:172.20.6.33, 1000.000000/1000.000000 bundle_group_0_7c45bd96450f2713ebbdfbc054264143}
In total there are 0 pending tasks and 1 pending actors on this node.
(pid=80782) 2021-10-14 09:40:26,614     WARNING backend_state.py:1072 -- Deployment 'A' has 1 replicas that have taken more than 30s to start up. This may be caused by waiting for the cluster to auto-scale or because the constructor is slow. Resources required for each replica: {'CPU': 1}, resources available: {'CPU': 7.0}. component=serve deployment=A
(pid=80782) 2021-10-14 09:40:56,690     WARNING backend_state.py:1072 -- Deployment 'A' has 1 replicas that have taken more than 30s to start up. This may be caused by waiting for the cluster to auto-scale or because the constructor is slow. Resources required for each replica: {'CPU': 1}, resources available: {'CPU': 7.0}. component=serve deployment=A
(pid=80782) 2021-10-14 09:41:26,741     WARNING backend_state.py:1072 -- Deployment 'A' has 1 replicas that have taken more than 30s to start u
p. This may be caused by waiting for the cluster to auto-scale or because the constructor is slow. Resources required for each replica: {'CPU':
 1}, resources available: {'CPU': 7.0}. component=serve deployment=A
(pid=80782) 2021-10-14 09:41:56,813     WARNING backend_state.py:1072 -- Deployment 'A' has 1 replicas that have taken more than 30s to start u
p. This may be caused by waiting for the cluster to auto-scale or because the constructor is slow. Resources required for each replica: {'CPU':
 1}, resources available: {'CPU': 7.0}. component=serve deployment=A
(pid=80782) 2021-10-14 09:42:26,910     WARNING backend_state.py:1072 -- Deployment 'A' has 1 replicas that have taken more than 30s to start u
p. This may be caused by waiting for the cluster to auto-scale or because the constructor is slow. Resources required for each replica: {'CPU':
 1}, resources available: {'CPU': 7.0}. component=serve deployment=A

@simon-mo
Copy link
Contributor

hint: just duplicate the ray minimal install tests and add some smart things to runtime env (serve inject ray serve into runtime env)

@triciasfu triciasfu changed the title Make sure Ray Serve installs ray[default] [nit] Make sure Ray Serve installs ray[default] Oct 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Request for new feature and/or capability good first issue Great starter issue for someone just starting to contribute to Ray P1 Issue that should be fixed within a few weeks serve Ray Serve Related Issue
Projects
None yet
3 participants