Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release test] [Cluster launcher] Add release test for aws example-full.yaml #34487

Merged
merged 3 commits into from
Apr 18, 2023
Merged

[Release test] [Cluster launcher] Add release test for aws example-full.yaml #34487

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8aba933
Add release test for aws `example-full.yaml`
architkulkarni Apr 17, 2023
802ab72
Add dashboard-host=0.0.0.0
architkulkarni Apr 17, 2023
6e12345
[bugfix] Update VolumeSize to integer
architkulkarni Apr 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions python/ray/autoscaler/aws/example-full.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ available_node_types:
BlockDeviceMappings:
- DeviceName: /dev/sda1
Ebs:
VolumeSize: 140GB
VolumeSize: 140
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This used to fail with

Traceback (most recent call last):
  File "/home/ray/anaconda3/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 2448, in main
    return cli()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py", line 856, in wrapper
    return f(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 1272, in up
    use_login_shells=use_login_shells,
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/commands.py", line 289, in create_or_update_cluster
    no_monitor_on_head,
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/commands.py", line 726, in get_or_create_head_node
    provider.create_node(head_node_config, head_node_tags, 1)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/aws/node_provider.py", line 320, in create_node
    created_nodes_dict = self._create_node(node_config, tags, count)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/aws/node_provider.py", line 424, in _create_node
    created = self.ec2_fail_fast.create_instances(**conf)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/boto3/resources/factory.py", line 580, in do_action
    response = action(self, *args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/botocore/client.py", line 924, in _make_api_call
    headers=additional_headers,
  File "/home/ray/anaconda3/lib/python3.7/site-packages/botocore/client.py", line 991, in _convert_to_request_dict
    api_params, operation_model
  File "/home/ray/anaconda3/lib/python3.7/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter BlockDeviceMappings[0].Ebs.VolumeSize, value: 140GB, type: <class 'str'>, valid types: <class 'int'>

# Additional options in the boto docs.
ray.worker.default:
# The minimum number of worker nodes of this type to launch.
Expand Down Expand Up @@ -169,7 +169,7 @@ worker_setup_commands: []
# Command to start ray on the head node. You don't need to change this.
head_start_ray_commands:
- ray stop
- ray start --head --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml
- ray start --head --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml --dashboard-host=0.0.0.0

# Command to start ray on worker nodes. You don't need to change this.
worker_start_ray_commands:
Expand Down
20 changes: 20 additions & 0 deletions release/release_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4812,3 +4812,23 @@
timeout: 1200
script: cd tests && python aws_launch_and_verify_cluster.py ../example-minimal.yaml
type: sdk_command

- name: aws_cluster_launcher_full
group: cluster-launcher-test
working_dir: ../python/ray/autoscaler/aws/

stable: true

# TODO: Migrate this test to Anyscale Jobs / staging_v2
env: prod_v1

frequency: nightly
team: core
cluster:
cluster_env: tests/aws_config.yaml
cluster_compute: tests/aws_compute.yaml

run:
timeout: 1200
script: cd tests && python aws_launch_and_verify_cluster.py ../example-full.yaml
type: sdk_command