Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-33913][SS] Upgrade Kafka to 2.8.0 #32325

Closed
wants to merge 1 commit into from
Closed

[SPARK-33913][SS] Upgrade Kafka to 2.8.0 #32325

wants to merge 1 commit into from

Conversation

dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Apr 24, 2021

What changes were proposed in this pull request?

This PR aims to upgrade Kafka client to 2.8.0.
Note that Kafka 2.8.0 uses ZSTD JNI 1.4.9-1 like Apache Spark 3.2.0.

Why are the changes needed?

This will bring the latest client-side improvement and bug fixes like the following examples.

  • KAFKA-10631 ProducerFencedException is not Handled on Offest Commit
  • KAFKA-10134 High CPU issue during rebalance in Kafka consumer after upgrading to 2.5
  • KAFKA-12193 Re-resolve IPs when a client is disconnected
  • KAFKA-10090 Misleading warnings: The configuration was supplied but isn't a known config
  • KAFKA-9263 The new hw is added to incorrect log when  ReplicaAlterLogDirsThread is replacing log
  • KAFKA-10607 Ensure the error counts contains the NONE
  • KAFKA-10458 Need a way to update quota for TokenBucket registered with Sensor
  • KAFKA-10503 MockProducer doesn't throw ClassCastException when no partition for topic

RELEASE NOTE

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs with the existing tests because this is a dependency change.

@SparkQA
Copy link

SparkQA commented Apr 24, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42420/

@SparkQA
Copy link

SparkQA commented Apr 24, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42420/

@SparkQA
Copy link

SparkQA commented Apr 25, 2021

Test build #137895 has finished for PR 32325 at commit 76ad092.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

Could you review this, @viirya ? This is for Structured Streaming and ZSTD dependency improvement.

Copy link
Member

@viirya viirya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks okay and CI already passed.

@HyukjinKwon
Copy link
Member

Merged to master.

@dongjoon-hyun
Copy link
Member Author

Thank you so much, @viirya and @HyukjinKwon !

@dongjoon-hyun dongjoon-hyun deleted the SPARK-33913 branch April 25, 2021 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants