Skip to content

Commit

Permalink
[SPARK-49569][BUILD][FOLLOWUP] Exclude spark-connect-shims from `sq…
Browse files Browse the repository at this point in the history
…l/core` module

### What changes were proposed in this pull request?
This pr exclude `spark-connect-shims` from `sql/core` module for further fix maven daily test.

### Why are the changes needed?
For fix maven daily test:

After #48399, although the Maven build was successful in my local environment, the Maven daily test pipeline still failed to build:

- https://github.com/apache/spark/actions/runs/11255598249/job/31311358712

```
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/ApproximatePercentileQuerySuite.scala:121: value makeRDD is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:75: value id is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.columnar.CachedBatch]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:82: value env is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:88: value env is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:185: value parallelize is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:481: value cleaner is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:500: value parallelize is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:940: value addSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:943: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:947: value removeSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1667: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1668: value addSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1673: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1674: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1682: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1683: value listenerBus is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1687: value removeSparkListener is not a member of org.apache.spark.SparkContext
Error: ] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala:1708: value partitions is not a member of org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
...
```

After using the `mvn dependency:tree` command to check, I found that `sql/core` cascadingly introduced `org.apache.spark:spark-connect-shims_2.13:jar:4.0.0-SNAPSHOT:test` through `org.apache.spark:spark-sql-api_2.13:test-jar:tests:4.0.0-SNAPSHOT:test`.

```
[INFO] ------------------< org.apache.spark:spark-sql_2.13 >-------------------
[INFO] Building Spark Project SQL 4.0.0-SNAPSHOT                        [18/42]
[INFO]   from sql/core/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- dependency:3.6.1:tree (default-cli)  spark-sql_2.13 ---
[INFO] org.apache.spark:spark-sql_2.13:jar:4.0.0-SNAPSHOT
...
[INFO] +- org.apache.spark:spark-catalyst_2.13:test-jar:tests:4.0.0-SNAPSHOT:test
[INFO] +- org.apache.spark:spark-sql-api_2.13:test-jar:tests:4.0.0-SNAPSHOT:test
[INFO] |  +- org.scala-lang.modules:scala-parser-combinators_2.13:jar:2.4.0:compile
[INFO] |  +- org.apache.spark:spark-connect-shims_2.13:jar:4.0.0-SNAPSHOT:test
[INFO] |  +- org.antlr:antlr4-runtime:jar:4.13.1:compile
[INFO] |  +- org.apache.arrow:arrow-vector:jar:17.0.0:compile
[INFO] |  |  +- org.apache.arrow:arrow-format:jar:17.0.0:compile
[INFO] |  |  +- org.apache.arrow:arrow-memory-core:jar:17.0.0:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.18.0:compile
[INFO] |  |  \- com.google.flatbuffers:flatbuffers-java:jar:24.3.25:compile
[INFO] |  \- org.apache.arrow:arrow-memory-netty:jar:17.0.0:compile
[INFO] |     \- org.apache.arrow:arrow-memory-netty-buffer-patch:jar:17.0.0:compile
```

This should be unexpected.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass GitHub Actions
- Pass maven test on GitHub Actions: https://github.com/LuciferYang/spark/runs/31314342332

<img width="996" alt="image" src="https://github.com/user-attachments/assets/6d2707f9-0d58-4c80-af27-cccdfa899e87">

All maven test passed

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #48403 from LuciferYang/test-maven-build.

Lead-authored-by: YangJie <[email protected]>
Co-authored-by: yangjie01 <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
  • Loading branch information
LuciferYang committed Oct 10, 2024
1 parent e589ccd commit b056e0b
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions sql/core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,12 @@
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-connect-shims_${scala.binary.version}</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down

0 comments on commit b056e0b

Please sign in to comment.