Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: convenience methods for reading and writing local files #1329

Merged
merged 4 commits into from
Sep 28, 2023
Merged

chore: convenience methods for reading and writing local files #1329

merged 4 commits into from
Sep 28, 2023

Conversation

eddyxu
Copy link
Contributor

@eddyxu eddyxu commented Sep 28, 2023 •
edited
Loading

Provide ObjectStore::create_local(std::path::Path) and open_local(std::path::Path)

@eddyxu eddyxu self-assigned this Sep 28, 2023
@westonpace
Copy link
Contributor

Is this mainly a convenience method for tests? Or are there other paths where we need this?

@eddyxu
Copy link
Contributor Author

eddyxu commented Sep 28, 2023

Is this mainly a convenience method for tests? Or are there other paths where we need this?

Shuffler (#1312) need. to use the local disk to shuffle IVF partitions to avoid OOM

westonpace
westonpace approved these changes Sep 28, 2023
View reviewed changes
Copy link
Contributor

@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe longer term we will want to have some kind of configurable "temporary storage" and this can evolve into that?

These changes seem fine though. One small minor nit that you can address or not.

rust/lance/src/io/object_writer.rs Outdated Show resolved Hide resolved
@eddyxu eddyxu merged commit 13e91c0 into main Sep 28, 2023
15 checks passed
@eddyxu eddyxu deleted the lei/local_tmp_rw branch September 28, 2023 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@westonpace westonpace westonpace approved these changes

@wjones127 wjones127 Awaiting requested review from wjones127

Assignees

@eddyxu eddyxu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@eddyxu @westonpace