Skip to content

Commit

Permalink
[SPARK-44115][BUILD] Upgrade Apache ORC to 2.0.0
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to Upgrade Apache ORC to 2.0.0 for Apache Spark 4.0.0.

Apache ORC community has 3-year support policy which is longer than Apache Spark. It's aligned like the following.
- Apache ORC 2.0.x <-> Apache Spark 4.0.x
- Apache ORC 1.9.x <-> Apache Spark 3.5.x
- Apache ORC 1.8.x <-> Apache Spark 3.4.x
- Apache ORC 1.7.x (Supported) <-> Apache Spark 3.3.x (End-Of-Support)

### Why are the changes needed?

**Release Note**
- https://github.com/apache/orc/releases/tag/v2.0.0

**Milestone**
- https://github.com/apache/orc/milestone/20?closed=1
  - apache/orc#1728
  - apache/orc#1801
  - apache/orc#1498
  - apache/orc#1627
  - apache/orc#1497
  - apache/orc#1509
  - apache/orc#1554
  - apache/orc#1708
  - apache/orc#1733
  - apache/orc#1760
  - apache/orc#1743

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#45443 from dongjoon-hyun/SPARK-44115.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
dongjoon-hyun committed Mar 8, 2024
1 parent 640ed4f commit 76b1c12
Show file tree
Hide file tree
Showing 3 changed files with 25 additions and 4 deletions.
7 changes: 4 additions & 3 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -227,9 +227,10 @@ opencsv/2.3//opencsv-2.3.jar
opentracing-api/0.33.0//opentracing-api-0.33.0.jar
opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
opentracing-util/0.33.0//opentracing-util-0.33.0.jar
orc-core/1.9.2/shaded-protobuf/orc-core-1.9.2-shaded-protobuf.jar
orc-mapreduce/1.9.2/shaded-protobuf/orc-mapreduce-1.9.2-shaded-protobuf.jar
orc-shims/1.9.2//orc-shims-1.9.2.jar
orc-core/2.0.0/shaded-protobuf/orc-core-2.0.0-shaded-protobuf.jar
orc-format/1.0.0/shaded-protobuf/orc-format-1.0.0-shaded-protobuf.jar
orc-mapreduce/2.0.0/shaded-protobuf/orc-mapreduce-2.0.0-shaded-protobuf.jar
orc-shims/2.0.0//orc-shims-2.0.0.jar
oro/2.0.8//oro-2.0.8.jar
osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
paranamer/2.8//paranamer-2.8.jar
Expand Down
17 changes: 16 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@
<!-- After 10.17.1.0, the minimum required version is JDK19 -->
<derby.version>10.16.1.1</derby.version>
<parquet.version>1.13.1</parquet.version>
<orc.version>1.9.2</orc.version>
<orc.version>2.0.0</orc.version>
<orc.classifier>shaded-protobuf</orc.classifier>
<jetty.version>11.0.20</jetty.version>
<jakartaservlet.version>5.0.0</jakartaservlet.version>
Expand Down Expand Up @@ -2593,13 +2593,28 @@
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-format</artifactId>
<version>1.0.0</version>
<classifier>${orc.classifier}</classifier>
<scope>${orc.deps.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
<version>${orc.version}</version>
<classifier>${orc.classifier}</classifier>
<scope>${orc.deps.scope}</scope>
<exclusions>
<exclusion>
<groupId>org.apache.orc</groupId>
<artifactId>orc-format</artifactId>
</exclusion>
<exclusion>
<groupId>com.aayushatharva.brotli4j</groupId>
<artifactId>brotli4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
Expand Down
5 changes: 5 additions & 0 deletions sql/core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@
<groupId>org.scala-lang.modules</groupId>
<artifactId>scala-parallel-collections_${scala.binary.version}</artifactId>
</dependency>
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-format</artifactId>
<classifier>${orc.classifier}</classifier>
</dependency>
<dependency>
<groupId>org.apache.orc</groupId>
<artifactId>orc-core</artifactId>
Expand Down

0 comments on commit 76b1c12

Please sign in to comment.