Provide option for max retires, and a call back when reached (potential DLQ)

[astubbs]

Atm, user has to implement the max retry functionality themselves, as documented here: retries docs branch

This should be part of the system, as an option for a max retry count. When the count is reached, a callback should be executed, providing the user the WorkContainer and the chance to do something with it - maybe produce it do a different topic (DLQ) or simply log it.

[sarwarbhuiyan]

Following up on the conversation we were having on this issue, there would be an approach here so as not to require the lambda signature to change to pass in WorkContainer vs ConsumerRecord.

Since the ConsumerRecord has a headers() object, we can increment a retry count as a known header in that in the framework. That way, the user could have access to the retryCount from the lambda and skip the exception (we could just do a docs section on that) if they want. Or the framework could compare a configured max_retries and check the retry_count header and trap out of continuing to process that record.

This is very similar to how Spring utilizes ConsumerRecord headers to pass a serialization exception so as to be able to handle a poison pill without causing an exception to be thrown.

[astubbs]

Draft implementation of this in #197 - user can easily implement max retries and DLQ upon exceeded

[astubbs]

FYI see PollContext object passed in to lambda to get failure counts of records.

[astubbs]

DLQ tracked here:

• Add a dead letter queue (DQL) implementation #310