producer.Send() blocks for too long time

[LvBay]

goroutine 102748631 [semacquire, 11167 minutes]:
sync.runtime_Semacquire(0xc026c4b9d8)
        /usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc026c4b9d0)
        /usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/apache/pulsar-client-go/pulsar.(*partitionProducer).Send(0xc005e5dd60, 0xe61760, 0xc00fc75080, 0xc0266fbc20, 0xbfaf54b1cfe715d9, 0x556c4fd0d57d, 0x1497240, 0xe61760)
        /Users/faker/go/pkg/mod/github.com/apache/[email protected]/pulsar/producer_partition.go:376 +0x171
github.com/apache/pulsar-client-go/pulsar.(*producer).Send(0xc0062611a0, 0xe61760, 0xc00fc75080, 0xc0266fbc20, 0xc00fc75080, 0xc01dc51d40, 0xc008babe10, 0x4055a5)

By visiting /debug/pprof/goroutine?debug=2 to get the running status of goutoutines, I found that some producer.Send() has been blocked and cannot be released.

Is there a way to solve it?

[jiazhai]

@wolfstudy Would you please help take a look

[merlimat]

@LvBay did you pass a ctx with a deadline timeout?

[LvBay]

        ...
	resp := make(chan struct{}, 1)
	var err error
	go func() {
		sctx, cancel := context.WithTimeout(context.Background(), time.Second)
		_, err = producer.Send(sctx, sendMsg)
		cancel()
		if err != nil {
			log.Error("publish message failed", err, msg)
		}
		resp <- struct{}{}
	}()
	select {
	case <-time.After(time.Second):
	case <-resp:
	}
        ...

Yeah, passed
@merlimat

[LvBay]

logrus log file shows something maybe help

time="2020-06-15T08:16:12+08:00" level=warning msg="Got unexpected send receipt for message: ledgerId:5194542 entryId:28 " local_addr="172.17.0.2:33614" producerID=1838 remote_addr="pulsar://xxx"

[merlimat]

@LvBay Do you have a way to repro this?

[liangyuanpeng]

I try to repro this and nothing 😶

[LvBay]

@merlimat @liangyuanpeng
I'm sorry I can't reproduce it in a test environment. But in a production environment, it will always appear at an irregular time.

What is even worse is that when the send timeout occurs, the producer of the partition will keep pushing the timeout.

What I can do is restart the program manually.

[LvBay]

I have fixed it. PR