Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit and integration tests for the vacuum command #227

Closed
rtyler opened this issue Apr 29, 2021 · 2 comments · May be fixed by #1817
Closed

Unit and integration tests for the vacuum command #227

rtyler opened this issue Apr 29, 2021 · 2 comments · May be fixed by #1817
Labels
binding/rust Issues for the Rust crate enhancement New feature or request good first issue Good for newcomers

Comments

@rtyler
Copy link
Member

rtyler commented Apr 29, 2021

Our preliminary support for vacuum would benefit from some more unit and integration tests to ensure that it's only deleting the right files whenever the vacuum is invoked.

@rtyler rtyler added enhancement New feature or request binding/rust Issues for the Rust crate labels Apr 29, 2021
@danmx
Copy link

danmx commented Sep 5, 2022

Referencing @mrk-its comment from #97 so it's not lost.

@rtyler @fvaleye It looks like there are still two serious issues with vacuum implementation:

* vacuum lists all files in dataset using `StorageBackend.list_objs`. The problem is this function returns all files (including these in subdirectories) on `s3` backend and `gcs` backend (althrough I'm not sure about gcs). On `file` and `azure` backends this function  lists only first-level files (without recursing to subdirectories).

* vacuum ignores files not referenced by delta log at all (so not included on DeltaTableState.files() and DeltaTableState.all_tombstones() lists).

@ion-elgreco
Copy link
Collaborator

This should be well tested at this point, also with introduction of VACUUM START, VACUUM END, I added a couple tests on the Python side

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/rust Issues for the Rust crate enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants