This repository has been archived by the owner on Jun 14, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 115
Add Alias column support in join condition for JoinIndex rule #25
Labels
Milestone
Comments
apoorvedave1
added
untriaged
This is the default tag for a newly created issue
enhancement
New feature or request
and removed
untriaged
This is the default tag for a newly created issue
labels
Jun 23, 2020
CC: @sezruby |
CC: @imback82 |
I think this should have been fixed by apache/spark#26943 in Spark 3.0, and I don't think it's worthwhile to update the rule to support this scenario for Spark 2.4. For the following: spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
val df1 = (0 until 100).map(i => (i % 5, i % 11)).toDF("i1", "j1")
val df2 = (0 until 100).map(i => (i % 7, i % 11)).toDF("i2", "j2")
df1.write.format("parquet").bucketBy(8, "i1").saveAsTable("t1")
df2.write.format("parquet").bucketBy(8, "i2").saveAsTable("t2")
sql("SELECT t1.i1, t2Temp.aliasC FROM t1 INNER JOIN (SELECT i2 as aliasC from t2) as t2Temp WHERE t1.i1 = t2Temp.aliasC").explain Spark 2.4.5 outputs:
whereas Spark 3.0 prints:
|
Great thanks! I'm closing this issue. |
@apoorvedave1 Can you confirm if this is not needed in the join rule? I see that we clean up the aliases, so matching should be ok, right? |
yes this looks like a solved issue now. Thanks for checking @imback82 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Describe the issue
Add support for
org.apache.spark.sql.catalyst.expressions.Alias
as column type inJoinIndexRule.scala
. Currently, this rule only supports Join Conditions onorg.apache.spark.sql.catalyst.expressions.Attribute
s . Consider a case where a columnc
from the base table has an aliasaliasC
. A Join query withc
will be able to utilize indexes but a query withaliasC
won't be.To Reproduce
Create a query plan of the form
Expected behavior
Query utilizes index
Screenshots
NA
Desktop (please complete the following information):
NA
Additional context
NA
The text was updated successfully, but these errors were encountered: