Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-enable RayDP tests since arrow version limits removed #31124

Merged
merged 11 commits into from
Feb 17, 2023
Merged

Re-enable RayDP tests since arrow version limits removed #31124

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
76ae690
uncomment raydp test
kira-lin Dec 15, 2022
a8b18ac
install java in ml docker image
kira-lin Jan 3, 2023
47b6255
install java in script
kira-lin Jan 29, 2023
ce25890
install raydp nightly
kira-lin Feb 13, 2023
b1e0685
address comments
kira-lin Feb 16, 2023
4ba2d2a
test build jar
kira-lin Feb 16, 2023
79cb22b
fix test
kira-lin Feb 16, 2023
322eb0d
fix
kira-lin Feb 17, 2023
e28cfd6
use install-java.sh
kira-lin Feb 17, 2023
31ace16
revert change
kira-lin Feb 17, 2023
1f8832c
Merge branch 'master' of https://github.com/ray-project/ray into fix-…
kira-lin Feb 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .buildkite/pipeline.ml.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,7 @@
instance_size: medium
commands:
- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/build/upload_build_info.sh; fi }; trap cleanup EXIT
- ./ci/env/install-java.sh
- DATA_PROCESSING_TESTING=1 ARROW_VERSION=nightly ./ci/env/install-dependencies.sh
- ./ci/env/env_info.sh
- bazel test --config=ci $(./ci/run/bazel_export_options) --build_tests_only --test_tag_filters=-dataset_integration python/ray/data/...
Expand All @@ -298,6 +299,7 @@
instance_size: medium
commands:
- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/build/upload_build_info.sh; fi }; trap cleanup EXIT
- ./ci/env/install-java.sh
- DATA_PROCESSING_TESTING=1 ARROW_VERSION=10.* ./ci/env/install-dependencies.sh
- ./ci/env/env_info.sh
- bazel test --config=ci $(./ci/run/bazel_export_options) --build_tests_only --test_tag_filters=-dataset_integration python/ray/data/...
Expand All @@ -308,6 +310,7 @@
instance_size: medium
commands:
- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/build/upload_build_info.sh; fi }; trap cleanup EXIT
- ./ci/env/install-java.sh
- DATA_PROCESSING_TESTING=1 ARROW_VERSION=9.* ./ci/env/install-dependencies.sh
- ./ci/env/env_info.sh
- bazel test --config=ci $(./ci/run/bazel_export_options) --build_tests_only --test_tag_filters=-dataset_integration python/ray/data/...
Expand All @@ -318,6 +321,7 @@
instance_size: medium
commands:
- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/build/upload_build_info.sh; fi }; trap cleanup EXIT
- ./ci/env/install-java.sh
- DATA_PROCESSING_TESTING=1 ARROW_VERSION=8.* ./ci/env/install-dependencies.sh
- ./ci/env/env_info.sh
- bazel test --config=ci $(./ci/run/bazel_export_options) --build_tests_only --test_tag_filters=-dataset_integration python/ray/data/...
Expand All @@ -328,6 +332,7 @@
instance_size: medium
commands:
- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/build/upload_build_info.sh; fi }; trap cleanup EXIT
- ./ci/env/install-java.sh
- DATA_PROCESSING_TESTING=1 ARROW_VERSION=7.* ./ci/env/install-dependencies.sh
- ./ci/env/env_info.sh
- bazel test --config=ci $(./ci/run/bazel_export_options) --build_tests_only --test_tag_filters=-dataset_integration python/ray/data/...
Expand All @@ -338,6 +343,7 @@
instance_size: medium
commands:
- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/build/upload_build_info.sh; fi }; trap cleanup EXIT
- ./ci/env/install-java.sh
- DATA_PROCESSING_TESTING=1 ARROW_VERSION=6.* ./ci/env/install-dependencies.sh
- ./ci/env/env_info.sh
- bazel test --config=ci $(./ci/run/bazel_export_options) --build_tests_only --test_tag_filters=-dataset_integration python/ray/data/...
Expand Down
15 changes: 15 additions & 0 deletions ci/env/install-java.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/usr/bin/env bash

# Push caller's shell options (quietly)
{ SHELLOPTS_STACK="${SHELLOPTS_STACK-}|$(set +o); set -$-"; } 2> /dev/null

set -euxo pipefail

SCRIPT_DIR=$(builtin cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd)
WORKSPACE_DIR="${SCRIPT_DIR}/../.."

sudo apt-get install -y maven openjdk-8-jre openjdk-8-jdk
"${WORKSPACE_DIR}"/java/build-jar-multiplatform.sh linux
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to build the Ray java jars? We don't run any java code, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raydp create java actors. Such as here


# Pop caller's shell options (quietly)
{ set -vx; eval "${SHELLOPTS_STACK##*|}"; SHELLOPTS_STACK="${SHELLOPTS_STACK%|*}"; } 2> /dev/null
8 changes: 1 addition & 7 deletions python/ray/data/tests/test_raydp_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
# Skipping entire test module:
# - Waiting for MLDataset to be made an optional dependency
# - See https://github.com/oap-project/raydp/pull/241

"""
import pytest
import ray
import raydp
Expand All @@ -12,7 +7,7 @@
@pytest.fixture(scope="function")
def spark(request):
ray.init(num_cpus=2, include_dashboard=False)
spark_session = raydp.init_spark("test", 1, 1, "500 M")
spark_session = raydp.init_spark("test", 1, 1, "500M")

def stop_all():
raydp.stop_spark()
Expand Down Expand Up @@ -59,4 +54,3 @@ def test_raydp_to_torch_iter(spark):
import sys

sys.exit(pytest.main(["-v", __file__]))
"""
5 changes: 1 addition & 4 deletions python/requirements/data_processing/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,6 @@ s3fs
modin>=0.8.3; python_version < '3.7'
modin>=0.11.0; python_version >= '3.7'
pytest-repeat
# Uncomment when RayDP:
# - Makes MLDataset an optional dependency.
# - Removes pyarrow < 7.0.0 upperbound.
# raydp-nightly==2022.6.30.dev1
raydp>=0.0.dev0
responses==0.13.4
pymars>=0.8.3