Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add checks in create/alter table if table location conflicts #7238

Closed
szehon-ho opened this issue Mar 29, 2023 · 4 comments
Closed

Add checks in create/alter table if table location conflicts #7238

szehon-ho opened this issue Mar 29, 2023 · 4 comments
Labels

Comments

@szehon-ho
Copy link
Collaborator

szehon-ho commented Mar 29, 2023

Feature Request / Improvement

Currently, when user makes multiple tables point to same location, running removeOrphanFiles on one of them will destroy the data of other tables.

Its a known issue, the the concept of table ownership is discussed at: #4159. Looks like there are two general directions, either

  1. The user explicitly chooses to share table locations among tables (to take more advantage of "write.object-storage.enabled"), and then instead of using table-level removeOrphanFiles, uses some warehouse-level remove orphan file (not currently in Iceberg OSS project)
  2. We discourage sharing of table locations for all other users.

This issue is to achieve the second one. Initial suggestions (can be changed)

  • Create/Alter table can check if there's other tables in catalog with the same location
  • Create/Alter table can check if there's any existing files in location.
  • This check can be configured at catalog level (?), in case the first option is achieved above and user wants to put tables in same location.

Though its not a perfect fix, it is a useful check to catch increasing problem, and it matches Trino-side fix, eg: trinodb/trino#12749

Query engine

None

@karuppayya
Copy link
Contributor

Thanks @szehon-ho for creating this issue, I will try to add some validations in the create/alter and remove orphan files flows.

@vamen
Copy link

vamen commented Apr 12, 2023

@karuppayya, have a similar issue. let me know if you have not started. will pick it up

@github-actions
Copy link

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

@github-actions github-actions bot added the stale label Oct 18, 2023
Copy link

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants