Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

…ctor_state

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Deletes the old IA.

Deleted all routes from old IA that were no longer used. (One remained: CmdResult)
Moved all the new IA routes from /new/<route> to /<route>
Deleted all components that were no longer used as a result of removing old IA routes
Deleted all parameters for "newIA" and removed the code paths where newIA is false
Deleted the dark theme from the the app (theme was only adjustable in the old IA)
I manually tested every single button to make sure links still worked. Our testing on the dashboard is not quite good enough yet (it's getting better) to trust the tests to catch all possible bugs here. The earlier we get this into nightly, the more manual testing from users we can get.

Signed-off-by: chaowang <[email protected]>

…project#31578)

Why are these changes needed?
Changes made to batch_tuning.ipynb and batch_forecasting.ipynb:

Update notebook texts, make steps clearer.
Tune outputs only showing SMOKE_TEST outputs.

---------

Signed-off-by: Christy Bergman <[email protected]>
Signed-off-by: Antoni Baum <[email protected]>
Co-authored-by: Antoni Baum <[email protected]>
Signed-off-by: chaowang <[email protected]>

…ct#33242)

Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Signed-off-by: chaowang <[email protected]>

Fix the test run.


Signed-off-by: chaowang <[email protected]>

70dbf41 pinned typeguard to make ax-platform work again on latest master, but a8a1ed0 updated ax-platform for python >= 3.8. The typeguard pin is incompatible with this pinned version. So we need to pin typeguard only for Python 3.7 to make the image builds for 3.8+ work again.

Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: chaowang <[email protected]>

…ject#33370)

The naming of different path-related components in Ray Tune is currently messy. For instance, `Experiment.local_dir` refers to main results directory, e.g. `~/ray_results`, while `Trial.local_dir` refers to the experiment directory, e.g. `~/ray_results/experiment`. The same is true for properties, where it's unclear if it refers to the object's sub directory or to its parent directory.

To disentangle this information, this PR introduces a new naming convention.

- All entities receiving a "parent path" receive it in a unambiguous naming scheme.
- For instance, `Experiment(storage_path)`, `TrialRunner(experiment_path)`, `Trial(experiment_path)`
- Outputs are also normalized. E.g. `Trial.remote_experiment_path` and `Trial.local_experiment_path`
- We keep existing arguments and properties for backwards compatibility

Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

…ct#33288)

Use SDK commands for all core tests.

It is because there was a big regression after migrating to V2 anyscale job runner.

Signed-off-by: chaowang <[email protected]>

…treaming execution (ray-project#33406)

Signed-off-by: chaowang <[email protected]>

Signed-off-by: Matthew Deng <[email protected]>
Signed-off-by: chaowang <[email protected]>

Signed-off-by: Avnish <[email protected]>
Signed-off-by: chaowang <[email protected]>

…api (ray-project#33316)

Signed-off-by: Avnish <[email protected]>
Signed-off-by: chaowang <[email protected]>

* Move adding params to learner hps to validate in order to be compatible with rllib yaml files
* Move learner_hp assignment from builder functions to validate

Signed-off-by: Avnish <[email protected]>
Signed-off-by: chaowang <[email protected]>

ray-project#33411 there is a risk ray client_server will crash with jsonschema >= 4.18


Signed-off-by: chaowang <[email protected]>

This PR adds 3.11 in the allowed runtime env conda version which was made available during the recent conda 2.23.0 release. conda/conda#11170 (comment)

It also fixes a a wheel rename for mac osx wheels from "*_intel.whl" to "*x86_64.whl" in recent builds.

A previous intel wheel build for mac:https://buildkite.com/ray-project/oss-ci-build-branch/builds/1268#0184a81c-6dd7-4cd2-8534-b6f763b8ce24
A more recent x86_64 name wheel build: https://buildkite.com/ray-project/oss-ci-build-branch/builds/2599#0186badf-4b95-44fd-b57d-6e42d5462c1d

Signed-off-by: rickyyx <[email protected]>
Co-authored-by: scv119 <[email protected]>
Signed-off-by: chaowang <[email protected]>

$ ./benchmark_worker_startup.py --help
usage: benchmark_worker_startup.py [-h] --num_cpus_in_cluster
                                   NUM_CPUS_IN_CLUSTER
                                   --num_tasks_or_actors_per_run
                                   NUM_TASKS_OR_ACTORS_PER_RUN
                                   --num_measurements_per_configuration
                                   NUM_MEASUREMENTS_PER_CONFIGURATION
This release test measures Ray worker startup time. Specifically, it measures
the time to start N different tasks or actors, where each task or actor imports
a large library (currently PyTorch). N is configurable. The test runs under a
few different configurations: {task, actor} x {runtime env, no runtime env} x
{GPU, no GPU} x {cold start, warm start}.
options:
  -h, --help            show this help message and exit
  --num_cpus_in_cluster NUM_CPUS_IN_CLUSTER
                        The number of CPUs in the cluster. This determines how
                        many CPU resources each actor/task requests.
  --num_tasks_or_actors_per_run NUM_TASKS_OR_ACTORS_PER_RUN
                        The number of tasks or actors per 'run'. A run starts
                        this many tasks/actors and consitutes a single
                        measurement. Several runs can be composed within a
                        single job for measure warm start, or spread across
                        different jobs to measure cold start.
  --num_measurements_per_configuration NUM_MEASUREMENTS_PER_CONFIGURATION
                        The number of measurements to record per configuration.
This script uses test_single_configuration.py to run the actual measurements.

Signed-off-by: chaowang <[email protected]>

This adds details about the live replicas for each deployment to be fetched from the new GET endpoint.

Sample replica detail from running a test application:
```
replica_id: app2_BasicDriver#KhlXQe
state: RUNNING
pid: 25853
actor_name: SERVE_REPLICA::app2_BasicDriver#KhlXQe
actor_id: 2355af670b023966af79501501000000
node_id: 3631e75fc5312752c54b567ee66491a1e58a0420f0abc5b1c44e70cf
node_ip: 192.168.0.141
start_time_s: 1678818083.039281
```

Details:
* `is_allocated` on each replica used to return just the node id for the controller to confirm the replica has been placed on a node and started. Now, it returns a tuple of runtime-context-related info:
  * `pid`
  * `actor_id`
  * `node_id`
  * `node_ip`
* The four fields listed above that are retrieved from the replica actor may be `None` before the actor is actually scheduled, so they are marked optional in the schema. (The rest of the fields are filled in immediately when the replica is created to be tracked in the controller)
```
class ReplicaDetails(BaseModel, extra=Extra.forbid):
    replica_id: str
    state: ReplicaState
    pid: Optional[int]
    actor_name: str
    actor_id: Optional[str]
    node_id: Optional[str]
    node_ip: Optional[str]
    start_time_s: float
```

Signed-off-by: chaowang <[email protected]>

…t#33340)

To simplify execution and not have to worry about covering all possible conflicts, we want to prevent users from deploying a single-app config (`ServeApplicationSchema`) first then switch to deploying a multi-app config (`ServeDeploySchema`), or vice versa.

Eventually we want to deprecate deploying using `ServeApplicationSchema`, so this also encourages users to migrate.

If users mix single-app and multi-app:
- we will raise an error in `controller.deploy_apps`
- the REST api will also return a `400 Response` with the error message

Signed-off-by: chaowang <[email protected]>

n this PR, we start the max_tasks_in_flight at 1 to make sure we fully utilize all the min_workers to start with. Then once min_workers have been created, we ramp up to the default max_tasks_in_flight of 4.

This allows us to maximize parallelism for datasets with a small number of blocks.

---------

Signed-off-by: amogkam <[email protected]>
Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>
Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

The current node option verification attempts to convert a string key to a dictionary when constructing an error message for blocked options, resulting in unclear / unintended exception.

Signed-off-by: dbczumar <[email protected]>
Signed-off-by: chaowang <[email protected]>

For long-living, memory leaking actors, it is more desirable to kill oldest task that is leaking the most. This avoid the situation where we constantly kill actor, which may lead to side effects where we generate a lot of log files, or trigger increased memory consumption in gcs / dashboard

Co-authored-by: Clarence Ng <[email protected]>
Signed-off-by: chaowang <[email protected]>

EBS is not necessary for many nodes test and it's 150GB by default. This reduce it to 30GB.
---------

Co-authored-by: Chen Shen <[email protected]>
Signed-off-by: chaowang <[email protected]>

Output from compare_perf_metrics:
https://docs.google.com/spreadsheets/d/1lSCLAQ7ghc4uOk80L3e9S2Rxuy7a2d3nbJRPi9D7brM/edit#gid=0

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

CPP generate-bazel-project-template is failing during build, due to extra space in the BUILD.bazel file. This commit fixes that.

Signed-off-by: Soumitra Kumar <[email protected]>
Signed-off-by: chaowang <[email protected]>

This PR restricts the number of profile events to be sent and aggregates task events from the same task attempt on the worker side to reduce the data sent to GCS.
This PR also refactos the metrics tracking to reduce lock contention on the core worker.

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

…t#32936)

Co-authored-by: angelinalg <[email protected]>
Co-authored-by: Justin Yu <[email protected]>
Co-authored-by: Yunxuan Xiao <[email protected]>
Co-authored-by: Yunxuan Xiao <[email protected]>
Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

Signed-off-by: chaowang <[email protected]>

…dev into cw/component_test_actors

…ent_test_actors