Feat/shared cache mk2 #581
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wip, because i have to run
Doesn't look like ops wants this and only introduces more complexity.
We used String or Path before to denote locations on the shared cache. Making this a specific key struct with info about what it contains is much more flexible and will allow us to move metrics etc to the right place.
Now that we have a SharedCacheKey we can easily do this and it makes more sense.
This results in some simplifications of the resulting code.
Now that symbolic has a ByteView::map_file_ref() constructor we can be a lot nicer on the filesystem and keep using filedescriptors to interact with the files.
just boring plumbing
Makes filesystem ops async as well, overlooked that before.
Swatinem
reviewed
Nov 23, 2021
added 3 commits
November 23, 2021 11:41
Plus avoid storing this in memory. We can't put several gigabytes into memory.
loewenheim
approved these changes
Nov 23, 2021
Swatinem
approved these changes
Nov 24, 2021
- Improve how we reopen the file. - Fix notfound on filesystem. - Tests test filedescriptor reading and writing in the same weird and wonderful ways as the Cacher and its CacheItemRequest users do.
These don't work yet, but probably due to incorrect test credentials.
Also fixup tests by making sure each GCS Object has a unique cache path.
That was pretty handwavy.
We use Body::wrap_stream to upload the body, so it is already streamed and I think this should be covered already.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Start of shared cache implementation.
TODO:
NATIVE-248