Add support for AtomicCreateTableAsSelect with Delta Lake [databricks] #9425
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jason Lowe <[email protected]>
|
build |
|
build |
|
build |
| @@ -92,6 +101,43 @@ object DeltaProviderImpl extends DeltaProviderImplBase { | |||
| val cpuFormat = format.asInstanceOf[DeltaParquetFileFormat] | |||
| GpuDeltaParquetFileFormat.convertToGpu(cpuFormat) | |||
| } | |||
|
|
|||
| override def isSupportedCatalog(catalogClass: Class[_ <: StagingTableCatalog]): Boolean = { | |||
| catalogClass == classOf[DeltaCatalog] || catalogClass == classOf[UnityCatalogV2Proxy] | |||
There was a problem hiding this comment.
question: are we guaranteed in this comparison that both catalogClass and classOf[DeltaCatalog] are loaded via the same ClassLoader instance, might get a false negative otherwise.
There was a problem hiding this comment.
I believe so. The technique essentially matches the technique used for almost all overrides, as GpuOverrides is building a large hashmap with classes as keys and doing a lookup on that map based on what we find in the CPU plan. So I would think that if we have an issue with this code, we'd have a similar issue with how all overrides work in the plugin.
revans2
approved these changes
Oct 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #7276. Extended ExternalSource to handle potential replacements for AtomicCreateTableAsSelect and added unit tests. A significant amount of the code change comes from the various GpuCreateDeltaTableCommand files which are small changes from the original versions of CreateDeltaTableCommand sources needed to perform the operation on the GPU. Rather than refactor these for commonality between the Delta Lake versions and make it hard to see the GPU diffs needed for each version, I added each separately to make diffing easier both for the current review and for auditing for changes in Delta Lake patch versions in the future.