[Enhancement] add output kafka support for zstd

[gwy1995]

Proposed commit message

add output kafka support for zstd

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[cla-checker-service]

❌ Author of the following commits did not sign a Contributor Agreement:
e9d618e, fce6f18, 0b8ab87, 2e7e563, 6424318, 543a348, f055b75, 7b1019f, , 9f91ab2, 3f9e577, ca805d9, 898c4bb, 057edee

Please, read and sign the above mentioned agreement if you want to contribute to this project

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @gwy1995? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[pierrehilbert]

Hello @gwy1995
Thanks for your contribution.
Could you please sign our CLA?

[gwy1995]

Hello @gwy1995 Thanks for your contribution. Could you please sign our CLA?

I have signed it for three times , but it still shows as unsigned in this issue. What should I do ?

[pierrehilbert]

Did you sign it with the same email than your github account?

[gwy1995]

Did you sign it with the same email than your github account?

yes, I have sign it today and yesterday for more than three times.

[faec]

Hi @gwy1995, we can try figuring out where the CLA checker is having trouble, but another issue I notice is that you removed Elastic's sarama fork and replaced it with an older (downgraded) upstream version. (See the readme in the fork for details.) The fork includes fixes for serious issues, so any changes to the sarama version should either update the fork's base version, or update to a mainline version that includes the fixes we need.

[gwy1995]

Hi @gwy1995, we can try figuring out where the CLA checker is having trouble, but another issue I notice is that you removed Elastic's sarama fork and replaced it with an older (downgraded) upstream version. (See the readme in the fork for details.) The fork includes fixes for serious issues, so any changes to the sarama version should either update the fork's base version, or update to a mainline version that includes the fixes we need.

Thanks for reminding me of this. I have rolled back the sarama version.
I found that the elastic version of sarama has merged the changes required by zstd, so just add one line of configuration. I wrote the details in the comments.Please take another look

[leehinman]

looks good. Any chance you could add a test case to

beats/libbeat/outputs/kafka/kafka_integration_test.go

Line 57 in 42dc965

func TestKafkaPublish(t *testing.T) {

?

[gwy1995]

looks good. Any chance you could add a test case to

beats/libbeat/outputs/kafka/kafka_integration_test.go

Line 57 in 42dc965

func TestKafkaPublish(t *testing.T) {

?

I add a test case. But I don't know why it didn't pass CI. I have tested it in the local environment.

[pierrehilbert]

/test