You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have rewritten many AQE test cases with hard-coded Spark configurations and judging conditions in each test case, because Gluten generates shuffle output of different sizes compared to the statistics used by AQE. Currently, the CH backend and the Velox backend share the Spark UT modules, thereby sharing the AQE test cases as well. Since different backends can generate varying shuffle statistics, we may need to separate the AQE Spark UTs. This separation will allow us to modify and use different AQE configurations for each backend individually.
So far, I found 2 test suites that need to be separated:
To address this issue, we can create packages org.apache.spark.sql.execution.adaptive.velox and org.apache.spark.sql.execution.adaptive.clickhouse and place the Suites for CH/Velox under these packages.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
We have rewritten many AQE test cases with hard-coded Spark configurations and judging conditions in each test case, because Gluten generates shuffle output of different sizes compared to the statistics used by AQE. Currently, the CH backend and the Velox backend share the Spark UT modules, thereby sharing the AQE test cases as well. Since different backends can generate varying shuffle statistics, we may need to separate the AQE Spark UTs. This separation will allow us to modify and use different AQE configurations for each backend individually.
So far, I found 2 test suites that need to be separated:
To address this issue, we can create packages
org.apache.spark.sql.execution.adaptive.velox
andorg.apache.spark.sql.execution.adaptive.clickhouse
and place the Suites for CH/Velox under these packages.Beta Was this translation helpful? Give feedback.
All reactions