Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an size bytes estimate for mongodb block #31930

Merged
merged 22 commits into from
Jan 26, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
edc51bd
Fix read_tfrecords_benchmark nightly test
jianoaix Dec 8, 2022
61f4d6d
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 14, 2022
a33a943
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 16, 2022
36ebe52
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 16, 2022
ce6763e
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 19, 2022
0e2c29e
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 21, 2022
f2b6ed0
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Dec 22, 2022
bb6c5c4
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 4, 2023
540fe79
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 10, 2023
edad7d0
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 10, 2023
60cc079
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 11, 2023
a3d3980
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 12, 2023
001579c
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 17, 2023
8aeed6c
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 18, 2023
7a9a49b
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 19, 2023
ef97167
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 20, 2023
6f0563c
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 21, 2023
bcec4d6
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 24, 2023
ddef4e5
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 25, 2023
fc9a175
Merge branch 'master' of https://github.com/ray-project/ray
jianoaix Jan 25, 2023
4e9ae0f
Provide an size bytes estimate for mongodb block
jianoaix Jan 25, 2023
b5f7a0c
Make sure datasource test is triggered for ray data code change
jianoaix Jan 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .buildkite/pipeline.ml.yml
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@
- bazel test --config=ci $(./ci/run/bazel_export_options) --build_tests_only --test_tag_filters=-client python/ray/util/dask/...

- label: ":potable_water: Dataset datasource integration tests (Python 3.7)"
conditions: ["NO_WHEELS_REQUIRED", "RAY_CI_PYTHON_AFFECTED"]
conditions: ["NO_WHEELS_REQUIRED", "RAY_CI_PYTHON_AFFECTED", "RAY_CI_DATA_AFFECTED"]
commands:
- cleanup() { if [ "${BUILDKITE_PULL_REQUEST}" = "false" ]; then ./ci/build/upload_build_info.sh; fi }; trap cleanup EXIT
- DATA_PROCESSING_TESTING=1 ARROW_VERSION=9.* ARROW_MONGO_VERSION=0.5.* ./ci/env/install-dependencies.sh
Expand Down
6 changes: 5 additions & 1 deletion python/ray/data/datasource/mongo_datasource.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,10 @@ def __init__(
self._client = pymongo.MongoClient(uri)
_validate_database_collection_exist(self._client, database, collection)

self._avg_obj_size = self._client[database].command("collstats", collection)[
"avgObjSize"
]

def estimate_inmemory_data_size(self) -> Optional[int]:
# TODO(jian): Add memory size estimation to improve auto-tune of parallelism.
return None
Expand Down Expand Up @@ -154,7 +158,7 @@ def make_block(
for i, partition in enumerate(partitions_ids):
metadata = BlockMetadata(
num_rows=partition["count"],
size_bytes=None,
size_bytes=partition["count"] * self._avg_obj_size,
schema=None,
input_files=None,
exec_stats=None,
Expand Down