[Bug] Dead Letter Policy in Consumer configuration doesn't account for chunking or batching configuration.

[jackson-chris]

Search before asking

• I searched in the issues and found nothing similar.

Version

2.11.1

Minimal reproduce step

• Produce a message to a topic that exceeds the maxMessageSize default value of 5MB.
• Create a consumer that consumes from that topic. This consumer should be configured with a deadLetterPolicy that includes retry letter and dead letter topics.
• Simulate a failure in the consumer code by forcing a call to reconsumeLater

What did you expect to see?

I would expect that a message will be produced to the retry letter topic that was configured in the deadLetterPolicy for that consumer.

What did you see instead?

The following error is observed:

org.apache.pulsar.client.api.PulsarClientException$InvalidMessageException: The producer mdl-pulsar-0-12 of the topic persistent://metadatalake/metadatalake/part-rlt-v1 sends a  message with 7094331 bytes that exceeds 5242880 bytes

Anything else?

The root cause is expected to be that the consumer implementation with respect to creating the producers for whatever is provided in deadLetterPolicy does not have any configuration parameters regarding if batching or chunking should be enabled. Problematic code:

• pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

  Lines 2092 to 2097 in ecd16d6

  	deadLetterProducer =
  	((ProducerBuilderImpl<byte[]>) client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema)))
  	.initialSubscriptionName(this.deadLetterPolicy.getInitialSubscriptionName())
  	.topic(this.deadLetterPolicy.getDeadLetterTopic())
  	.blockIfQueueFull(false)
  	.createAsync();

• pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

  Lines 613 to 617 in ecd16d6

  	retryLetterProducer = client.newProducer(Schema.AUTO_PRODUCE_BYTES(schema))
  	.topic(this.deadLetterPolicy.getRetryLetterTopic())
  	.enableBatching(false)
  	.blockIfQueueFull(false)
  	.create();

Are you willing to submit a PR?

• I'm willing to submit a PR!