Enabling idempotency still allows producing same message

[bentcoder]

Description

Hi,

Maybe I am getting the meaning/aim of idempotency in this library or Kafka in general. However, what I understand from the docs, code comments and other blog posts is that, enabling it (like show below) is meant to produce same message exactly once. Correct me if I am wrong please.

Technically, I am trying to produce same message exactly once.

Thanks

Note: I have seen this and this examples but they seem very complicated rather than keeping the producer and consumer in different files. To be honest, I am not even sure if they are applicable to me.

Versions

Sarama	Kafka	Go
v1.43.2	v3.3.2	1.22

Configuration

package main

import (
	"log"
	"time"

	"github.com/IBM/sarama"
)

func main() {
	cfg := sarama.NewConfig()
	cfg.ClientID = "test"
	cfg.Version = sarama.V3_3_2_0
	cfg.Producer.Return.Successes = true
	cfg.Producer.Retry.Max = 30
	cfg.Producer.Retry.Backoff = time.Millisecond * 10

	// Enable EOS (Exactly Once Semantic)
	cfg.Producer.Idempotent = true
	cfg.Producer.RequiredAcks = sarama.WaitForAll
	cfg.Net.MaxOpenRequests = 1
	//

	pro, err := sarama.NewSyncProducer([]string{":9093"}, cfg)
	if err != nil {
		log.Fatalln(err)
	}

	msg := sarama.ProducerMessage{
		Topic:     "warnings-topic", // This topic has only one partition
		Value:     sarama.StringEncoder("body"),
		Timestamp: time.Now(),
	}

	// Purposely simulating duplicate messages
	for i := 1; i <= 2; i++ {
		prt, off, err := pro.SendMessage(&msg)
		if err != nil {
			log.Fatalln(err)
		}

		log.Printf("I: %d PARTITION: %d OFFSET: %d\n", i, prt, off)
	}
}

$ go run -race main.go
2024/05/19 19:22:21 I: 1PARTITION: 0 OFFSET: 39841
2024/05/19 19:22:21 I: 2PARTITION: 0 OFFSET: 39842

Additional Context

[dnwe]

👋🏻 thanks for getting in touch. Indeed the Kafka Producer Idempotency specifically relates to the built-in retry mechanism and de-duplication of those — not that it will do de-duplication of messages that you ask the client to send.

So the intention is that if you enable Producer.Idempotent then you can technically set cfg.Producer.Retry.Max to math.MaxInt and still have a guarantee that even during network turbulence or broker restarts you'll only end up with exactly one copy of your message stored in Kafka

[bentcoder]

Thank you for the response.

Indeed the Kafka Producer Idempotency specifically relates to the built-in retry mechanism and de-duplication of those — not that it will do de-duplication of messages that you ask the client to send.

I see, got it thank you ...

So the intention is that if you enable Producer.Idempotent then you can technically set cfg.Producer.Retry.Max to math.MaxInt and still have a guarantee that even during network turbulence or broker restarts you'll only end up with exactly one copy of your message stored in Kafka

But if I get it right this proposition (cfg.Producer.Retry.Max = 1) would be counterproductive and beat the purpose of retrying mechanism. If a message is not sent for any failure at first instance, no more tries hence the message is permanently lost. I am not missing anything, am I?