AWS S3 IO through KvikIO #16499
Open
AWS S3 IO through KvikIO #16499
+144
−15
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
madsbk
added
improvement
github-actions
bot
added
libcudf
madsbk
force-pushed
the
kvikio-remote-io
branch
2 times, most recently
from
dd1b46d
to
55cc220
Compare
madsbk
force-pushed
the
kvikio-remote-io
branch
from
55cc220
to
02c19cc
Compare
github-actions
bot
removed
the
pylibcudf
madsbk
force-pushed
the
kvikio-remote-io
branch
from
55d817f
to
7d2a170
Compare
madsbk
force-pushed
the
kvikio-remote-io
branch
from
41ae12e
to
f4e0f57
Compare
github-actions
bot
added
pylibcudf
madsbk
force-pushed
the
kvikio-remote-io
branch
2 times, most recently
from
ad2e46d
to
ea983e1
Compare
madsbk
force-pushed
the
kvikio-remote-io
branch
2 times, most recently
from
4f94346
to
22c0222
Compare
madsbk
force-pushed
the
kvikio-remote-io
branch
3 times, most recently
from
d8e0c8d
to
a1a7ee1
Compare
madsbk
force-pushed
the
kvikio-remote-io
branch
3 times, most recently
from
50f30c2
to
f6c710a
Compare
madsbk
force-pushed
the
kvikio-remote-io
branch
from
37717e7
to
167c5d2
Compare
madsbk
requested review from
isVoid,
Matt711,
shrshi,
pmattione-nvidia and
vuule
rjzamora
reviewed
Oct 24, 2024
cpp/src/io/utilities/datasource.cpp
Outdated
| */ | ||
| static bool is_supported_remote_url(std::string const& url) | ||
| { | ||
| // Regular expression to match "<s3|http|https>://" |
There was a problem hiding this comment.
Wondering about long-term: If we're given an http or https URI, how would we tell whether that's on amazon s3 vs some other cloud service?
There was a problem hiding this comment.
E.g. if users rely on https automatically working for s3, then we need to change how https works in the future, could their code break.
There was a problem hiding this comment.
Good point, I have limited the support to s3:// urls for now.
KvikIO also supports regular http servers but let's address that in a later PR.
vuule
reviewed
Oct 25, 2024
Co-authored-by: Vukasin Milovanovic <[email protected]>
pmattione-nvidia
approved these changes
Oct 28, 2024
vuule
approved these changes
Oct 28, 2024
shrshi
approved these changes
Oct 29, 2024
| CUDF_EXPECTS(supports_device_read(), "Device reads are not supported for this file."); | ||
|
|
||
| auto const read_size = std::min(size, this->size() - offset); | ||
| return _kvikio_file.pread(dst, read_size, offset); |
There was a problem hiding this comment.
Question: If insufficient memory is allocated in dst, does pread catch the exception thrown?
This was referenced Oct 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement remote IO read using KvikIO's S3 backend.
For now, this is an experimental feature for parquet read only. Enable by defining
CUDF_KVIKIO_REMOTE_IO=ON.Checklist