HIVE-28536: Iceberg: support for assigning manual compaction requests on custom compaction pools #5503
+92
−41
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
difin
force-pushed
the
iceberg_custom_pool
branch
from
a69ebc9
to
5729291
Compare
asf-ci-hive
added
tests failed
tests pending
tests unstable
and removed
tests pending
tests failed
labels
difin
force-pushed
the
iceberg_custom_pool
branch
from
5729291
to
447150a
Compare
|
deniskuzZ
reviewed
Oct 14, 2024
| @@ -2234,7 +2234,9 @@ public static enum ConfVars { | |||
| "If this is set to true the URI for auth will have the default location masked with DEFAULT_TABLE_LOCATION"), | |||
| HIVE_ICEBERG_ALLOW_DATAFILES_IN_TABLE_LOCATION_ONLY("hive.iceberg.allow.datafiles.in.table.location.only", false, | |||
| "If this is set to true, then all the data files being read should be withing the table location"), | |||
|
|
|||
| HIVE_ICEBERG_COMPACTOR_POOL_NAME("hive.iceberg.compactor.pool.name", "default", | |||
There was a problem hiding this comment.
let's not create Hive config for that. can we use tbl properties instead?
reg the default value, maybe iceberg-default
There was a problem hiding this comment.
Fixed. Now if the pool name is not provided in the compaction command it attempts to get it from the table property hive.compactor.worker.pool, otherwise compacts on the default pool.
difin
changed the title
difin
force-pushed
the
iceberg_custom_pool
branch
from
447150a
to
d8d47f7
Compare
difin
force-pushed
the
iceberg_custom_pool
branch
from
d8d47f7
to
2aad09f
Compare
…on on a custom compaction pool.
difin
force-pushed
the
iceberg_custom_pool
branch
from
2aad09f
to
2110fc7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Support for assigning manual compaction requests on custom compaction pools by using table property
hive.compactor.worker.pool.Why are the changes needed?
On downstream environments sometimes a single HMS is used with several HS2s with enabled compaction workers, where some HS2s are running on Hive code versions that don't have Iceberg compaction functionality. It is needed to prevent these HS2s from attempting to process Iceberg compaction requests. This PR enables solving this problem using custom compaction pooling.
Does this PR introduce any user-facing change?
Yes, it adds the ability of compacting on custom compaction pool whose name is specified in a table property when processing manual compaction requests.
Is the change a dependency upgrade?
No
How was this patch tested?
Updated q-tests