Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add handling for unmanaged files to vacuum command #1817

Open
wants to merge 25 commits into
base: main
Choose a base branch
from

Commits on Nov 7, 2023

  1. Configuration menu
    Copy the full SHA
    1dc65b3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e59bb34 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. chore: upgrade to the latest dynamodb-lock crate

    The new version of this crate properly sets a lease duration such that the locks
    can actually expire
    rtyler authored and Jan-Schweizer committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    90b7741 View commit details
    Browse the repository at this point in the history
  2. feat: default logstore implementation (delta-io#1742)

    # Description
    
    Introduce a `LogStore` abstraction to channel all log store reads and
    writes through a single place. This is supposed to allow implementations
    with more sophisticated locking mechanisms that do not rely on atomic
    rename semantics for the underlying object store.
    
    This does not change any functionality - it reorganizes read operations
    and commits on the delta commit log to be funneled through the
    respective methods of `LogStore`.
    
    ## Rationale
    
    The goal is to align the implementation of multi-cluster writes for
    Delta Lake on S3 with the one provided by the original `delta` library,
    enabling multi-cluster writes with some writers using Spark / Delta
    library and other writers using `delta-rs` For an overview of how it's
    done in delta, please see:
    1. Delta [blog
    post](https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3/)
    (high-level concept)
    2. Associated Databricks [design
    doc](https://docs.google.com/document/d/1Gs4ZsTH19lMxth4BSdwlWjUNR-XhKHicDvBjd2RqNd8/edit#heading=h.mjjuxw9mcz9h)
    (detailed read)
    3.
    [S3DynamoDbLogStore.java](https://github.com/delta-io/delta/blob/master/storage-s3-dynamodb/src/main/java/io/delta/storage/S3DynamoDBLogStore.java)(content
    warning: Java code behind this link)
    
    This approach requires readers of a delta table to "recover" unfinished
    commits from writers - as a result, reading and writing is combined in a
    single interface, which in this PR is modeled after
    [LogStore.java](https://github.com/delta-io/delta/blob/master/storage/src/main/java/io/delta/storage/LogStore.java).
    Currently in `delta-rs`, read path for commits is implemented directly
    in `DeltaTable`, and there's no mechanism to implement storage-specific
    behavior like interacting with DynamoDb.
    
    ---------
    
    Co-authored-by: Robert Pack <[email protected]>
    2 people authored and Jan-Schweizer committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    809f645 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8e64a0d View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. Configuration menu
    Copy the full SHA
    7559c24 View commit details
    Browse the repository at this point in the history
  2. fix: run integration tests in CI

    roeap authored and Jan-Schweizer committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    da6e438 View commit details
    Browse the repository at this point in the history
  3. Update README.md

    Update the Slack link to https://go.delta.io/slack
    dennyglee authored and Jan-Schweizer committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    140f949 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2023

  1. Configuration menu
    Copy the full SHA
    a327fa8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b40f276 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2023

  1. Configuration menu
    Copy the full SHA
    a358f06 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    752773a View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. Configuration menu
    Copy the full SHA
    dc74dcc View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2023

  1. Configuration menu
    Copy the full SHA
    96a5e0a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5ea77fd View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2023

  1. docs: fix all examples and change overall structure (delta-io#1931)

    # Description
    I have made a bunch of improvements to fix the overall structure due to
    example sections not being consistent. I've also enabled some extra
    features. Fixed also the issue of some classes/functions not being shown
    properly.
    ion-elgreco authored and Jan-Schweizer committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    48b4e3c View commit details
    Browse the repository at this point in the history
  2. fix: prune each merge bin with only 1 file (delta-io#1902)

    # Description
    This PR prunes each merge bin with only 1 file, even though there are
    multiple merge bins.
    
    # Related Issue(s)
    - closes delta-io#1901 
    
    # Documentation
    This PR adds test_idempotent_with_multiple_bins() for testing.
    haruband authored and Jan-Schweizer committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    f90b48c View commit details
    Browse the repository at this point in the history
  3. chore: update python version (delta-io#1934)

    # Description
    Prepare for next release
    
    # Related Issue(s)
    <!---
    For example:
    
    - closes delta-io#106
    --->
    
    # Documentation
    
    <!---
    Share links to useful documentation
    --->
    wjones127 authored and Jan-Schweizer committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    d518f40 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2733f3d View commit details
    Browse the repository at this point in the history
  5. add type param

    wjones127 authored and Jan-Schweizer committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    f4b9e91 View commit details
    Browse the repository at this point in the history
  6. fix: get rid of panic in during table (delta-io#1928)

    # Description
    This is a continuation of the discussion in the
    delta-io#1917 Getting rid of panic in
    the library crate in favor of returning an error so lib users could
    handle it in a way they see it Test changes accordingly
    
    Co-authored-by: Robert Pack <[email protected]>
    2 people authored and Jan-Schweizer committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    83f2f99 View commit details
    Browse the repository at this point in the history
  7. Happify linter

    Jan-Schweizer committed Dec 2, 2023
    Configuration menu
    Copy the full SHA
    b946d07 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    d4642bf View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2023

  1. Configuration menu
    Copy the full SHA
    04e3811 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2023

  1. Configuration menu
    Copy the full SHA
    27c2b53 View commit details
    Browse the repository at this point in the history