Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-30552][Connector/Pulsar] drop batch message size assertion, better set the cursor position. #11

Merged
merged 1 commit into from
Jan 5, 2023
Merged

[FLINK-30552][Connector/Pulsar] drop batch message size assertion, better set the cursor position. #11

merged 1 commit into from
Jan 5, 2023

Conversation

syhily
Copy link
Contributor

@syhily syhily commented Jan 3, 2023 •
edited
Loading

This PR fixes three things.

  1. Drop the batch size assertion, the batch size is based on the producer which can't promise the size is 1.
  2. Drop the next message id calculation.
  3. Use Admin.topics().resetCursor() api to set the consuming position. No need to calculation the next message id.
  4. Support using the ChunckMessageIdImpl

@syhily
Copy link
Contributor Author

syhily commented Jan 3, 2023

The Pulsar's PIP-229 introduce the new interface for exposing the MessageIdData which can reduce a lot of boilerplate codes. This PR only gets the bug fixed. We can better solve it in the future.

@syhily syhily force-pushed the hotfix/wrong-batch-message-id branch 2 times, most recently from 5cb9340 to 4cac626 Compare January 3, 2023 14:17
@syhily syhily changed the title [FLINK-30552][Connector/Pulsar] Drop batch message size assertion, fix the next message id calculation. [FLINK-30552][Connector/Pulsar] Drop batch message size assertion, better set the cursor position. Jan 3, 2023
@syhily syhily changed the title [FLINK-30552][Connector/Pulsar] Drop batch message size assertion, better set the cursor position. [FLINK-30552][Connector/Pulsar] drop batch message size assertion, better set the cursor position. Jan 3, 2023
@syhily syhily force-pushed the hotfix/wrong-batch-message-id branch 2 times, most recently from 391fe7d to 0bc6696 Compare January 3, 2023 14:26
@syhily
[FLINK-30552][Connector/Pulsar] Drop next message id calculation, use…
1044b8b 
… resetCursor api to exclude the given message.
@syhily syhily force-pushed the hotfix/wrong-batch-message-id branch from 0bc6696 to 1044b8b Compare January 3, 2023 14:26
tisonkun
tisonkun approved these changes Jan 5, 2023
View reviewed changes
Copy link
Member

@tisonkun tisonkun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Merging...

@tisonkun tisonkun merged commit 6e5ed0c into apache:main Jan 5, 2023
@syhily syhily deleted the hotfix/wrong-batch-message-id branch January 5, 2023 05:59
@syhily syhily mentioned this pull request Jan 10, 2023
Merged
cbornet pushed a commit to cbornet/flink-connector-pulsar that referenced this pull request Jun 12, 2023
@syhily @cbornet
[FLINK-30552][Connector/Pulsar] Drop next message id calculation, use…
e9812b7 
… resetCursor api to exclude the given message. (apache#11)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tisonkun tisonkun tisonkun approved these changes

Assignees
No one assigned
Labels
component=Connectors/Pulsar
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@syhily @tisonkun