-
Notifications
You must be signed in to change notification settings - Fork 407
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: add handling for unmanaged files to vacuum command #1817
base: main
Are you sure you want to change the base?
fix: add handling for unmanaged files to vacuum command #1817
Commits on Nov 7, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 1dc65b3 - Browse repository at this point
Copy the full SHA 1dc65b3View commit details -
Configuration menu - View commit details
-
Copy full SHA for e59bb34 - Browse repository at this point
Copy the full SHA e59bb34View commit details
Commits on Nov 9, 2023
-
chore: upgrade to the latest dynamodb-lock crate
The new version of this crate properly sets a lease duration such that the locks can actually expire
Configuration menu - View commit details
-
Copy full SHA for 90b7741 - Browse repository at this point
Copy the full SHA 90b7741View commit details -
feat: default logstore implementation (delta-io#1742)
# Description Introduce a `LogStore` abstraction to channel all log store reads and writes through a single place. This is supposed to allow implementations with more sophisticated locking mechanisms that do not rely on atomic rename semantics for the underlying object store. This does not change any functionality - it reorganizes read operations and commits on the delta commit log to be funneled through the respective methods of `LogStore`. ## Rationale The goal is to align the implementation of multi-cluster writes for Delta Lake on S3 with the one provided by the original `delta` library, enabling multi-cluster writes with some writers using Spark / Delta library and other writers using `delta-rs` For an overview of how it's done in delta, please see: 1. Delta [blog post](https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3/) (high-level concept) 2. Associated Databricks [design doc](https://docs.google.com/document/d/1Gs4ZsTH19lMxth4BSdwlWjUNR-XhKHicDvBjd2RqNd8/edit#heading=h.mjjuxw9mcz9h) (detailed read) 3. [S3DynamoDbLogStore.java](https://github.com/delta-io/delta/blob/master/storage-s3-dynamodb/src/main/java/io/delta/storage/S3DynamoDBLogStore.java)(content warning: Java code behind this link) This approach requires readers of a delta table to "recover" unfinished commits from writers - as a result, reading and writing is combined in a single interface, which in this PR is modeled after [LogStore.java](https://github.com/delta-io/delta/blob/master/storage/src/main/java/io/delta/storage/LogStore.java). Currently in `delta-rs`, read path for commits is implemented directly in `DeltaTable`, and there's no mechanism to implement storage-specific behavior like interacting with DynamoDb. --------- Co-authored-by: Robert Pack <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 809f645 - Browse repository at this point
Copy the full SHA 809f645View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8e64a0d - Browse repository at this point
Copy the full SHA 8e64a0dView commit details
Commits on Nov 10, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 7559c24 - Browse repository at this point
Copy the full SHA 7559c24View commit details -
Configuration menu - View commit details
-
Copy full SHA for da6e438 - Browse repository at this point
Copy the full SHA da6e438View commit details -
Update the Slack link to https://go.delta.io/slack
Configuration menu - View commit details
-
Copy full SHA for 140f949 - Browse repository at this point
Copy the full SHA 140f949View commit details
Commits on Nov 11, 2023
-
Correctly handle hidden files in _change_data and _delta_index & dele…
…tion vector files
Configuration menu - View commit details
-
Copy full SHA for a327fa8 - Browse repository at this point
Copy the full SHA a327fa8View commit details -
Configuration menu - View commit details
-
Copy full SHA for b40f276 - Browse repository at this point
Copy the full SHA b40f276View commit details
Commits on Nov 15, 2023
-
Configuration menu - View commit details
-
Copy full SHA for a358f06 - Browse repository at this point
Copy the full SHA a358f06View commit details -
Configuration menu - View commit details
-
Copy full SHA for 752773a - Browse repository at this point
Copy the full SHA 752773aView commit details
Commits on Nov 29, 2023
-
Configuration menu - View commit details
-
Copy full SHA for dc74dcc - Browse repository at this point
Copy the full SHA dc74dccView commit details
Commits on Nov 30, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 96a5e0a - Browse repository at this point
Copy the full SHA 96a5e0aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5ea77fd - Browse repository at this point
Copy the full SHA 5ea77fdView commit details
Commits on Dec 2, 2023
-
docs: fix all examples and change overall structure (delta-io#1931)
# Description I have made a bunch of improvements to fix the overall structure due to example sections not being consistent. I've also enabled some extra features. Fixed also the issue of some classes/functions not being shown properly.
Configuration menu - View commit details
-
Copy full SHA for 48b4e3c - Browse repository at this point
Copy the full SHA 48b4e3cView commit details -
fix: prune each merge bin with only 1 file (delta-io#1902)
# Description This PR prunes each merge bin with only 1 file, even though there are multiple merge bins. # Related Issue(s) - closes delta-io#1901 # Documentation This PR adds test_idempotent_with_multiple_bins() for testing.
Configuration menu - View commit details
-
Copy full SHA for f90b48c - Browse repository at this point
Copy the full SHA f90b48cView commit details -
chore: update python version (delta-io#1934)
# Description Prepare for next release # Related Issue(s) <!--- For example: - closes delta-io#106 ---> # Documentation <!--- Share links to useful documentation --->
Configuration menu - View commit details
-
Copy full SHA for d518f40 - Browse repository at this point
Copy the full SHA d518f40View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2733f3d - Browse repository at this point
Copy the full SHA 2733f3dView commit details -
Configuration menu - View commit details
-
Copy full SHA for f4b9e91 - Browse repository at this point
Copy the full SHA f4b9e91View commit details -
fix: get rid of panic in during table (delta-io#1928)
# Description This is a continuation of the discussion in the delta-io#1917 Getting rid of panic in the library crate in favor of returning an error so lib users could handle it in a way they see it Test changes accordingly Co-authored-by: Robert Pack <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 83f2f99 - Browse repository at this point
Copy the full SHA 83f2f99View commit details -
Configuration menu - View commit details
-
Copy full SHA for b946d07 - Browse repository at this point
Copy the full SHA b946d07View commit details -
Configuration menu - View commit details
-
Copy full SHA for d4642bf - Browse repository at this point
Copy the full SHA d4642bfView commit details
Commits on Dec 14, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 04e3811 - Browse repository at this point
Copy the full SHA 04e3811View commit details
Commits on Dec 20, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 27c2b53 - Browse repository at this point
Copy the full SHA 27c2b53View commit details