-
Notifications
You must be signed in to change notification settings - Fork 25
Bump pulsar version to 3.1.0-SNAPSHOT #5894
Commits on Aug 29, 2023
-
[fix][client] Fix consumer can't consume resent chunked messages (apa…
…che#21070) ### Motivation Current, when the producer resend the chunked message like this: - M1: UUID: 0, ChunkID: 0 - M2: UUID: 0, ChunkID: 0 // Resend the first chunk - M3: UUID: 0, ChunkID: 1 When the consumer received the M2, it will find that it's already tracking the UUID:0 chunked messages, and will then discard the message M1 and M2. This will lead to unable to consume the whole chunked message even though it's already persisted in the Pulsar topic. Here is the code logic: https://github.com/apache/pulsar/blob/44a055b8a55078bcf93f4904991598541aa6c1ee/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1436-L1482 The bug can be easily reproduced using the testcase `testResendChunkMessages` introduced by this PR. ### Modifications - When receiving the new duplicated first chunk of a chunked message, the consumer discard the current chunked message context and create a new context to track the following messages. For the case mentioned in Motivation, the M1 will be released and the consumer will assemble M2 and M3 as the chunked message.
Configuration menu - View commit details
-
Copy full SHA for eb2e3a2 - Browse repository at this point
Copy the full SHA eb2e3a2View commit details
Commits on Aug 30, 2023
-
Configuration menu - View commit details
-
Copy full SHA for eded9f1 - Browse repository at this point
Copy the full SHA eded9f1View commit details -
[fix][build] Upgrade Guava to 32.1.2-jre (apache#21090)
Masahiro Sakamoto authoredAug 30, 2023 Configuration menu - View commit details
-
Copy full SHA for dab5b2f - Browse repository at this point
Copy the full SHA dab5b2fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 53ffe81 - Browse repository at this point
Copy the full SHA 53ffe81View commit details -
Configuration menu - View commit details
-
Copy full SHA for 99e3fea - Browse repository at this point
Copy the full SHA 99e3feaView commit details
Commits on Aug 31, 2023
-
[fix] [broker] remove bundle-data in local metadata store. (apache#21078
Configuration menu - View commit details
-
Copy full SHA for 4a87c64 - Browse repository at this point
Copy the full SHA 4a87c64View commit details -
[fix][broker]Fix chunked messages will be filtered by duplicating (ap…
…ache#20948) ## Motivation Make the chunk message function work properly when deduplication is enabled. ## Modification ### Only check and store the sequence ID of the last chunk in a chunk message. For example: ```markdown Chunk-1 sequence ID: 0, chunk ID: 0, total chunk: 2 Chunk-2 sequence ID: 0, chunk ID: 1 Chunk-3 sequence ID: 1, chunk ID: 0 total chunk: 3 Chunk-4 sequence ID: 1, chunk ID: 1 Chunk-5 sequence ID: 1, chunk ID: 1 Chunk-6 sequence ID: 1, chunk ID: 2 ``` Only store check and store the sequence ID of Chunk-2 and Chunk-6. **Add a property in the publishContext to determine whether this chunk is the last chunk when persistent completely.** ```java publishContext.setProperty(IS_LAST_CHUNK, Boolean.FALSE); ``` ### Filter and ack duplicated chunks in a chunk message instead of discarding ctx. For example: ```markdown Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1 Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2 Chunk-3 sequence ID: 0, chunk ID: 2, msgID: 1:3 Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4 Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5 Chunk-6 sequence ID: 0, chunk ID: 3, msgID: 1:6 ``` We should filter and ack chunk-4 and chunk-5.
Configuration menu - View commit details
-
Copy full SHA for b0b13bc - Browse repository at this point
Copy the full SHA b0b13bcView commit details -
[fix][io] Allow setting sourceType in config file (apache#19836)
Signed-off-by: tison <[email protected]> Co-authored-by: Alexander Preuß <[email protected]> Co-authored-by: tison <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for f1c8684 - Browse repository at this point
Copy the full SHA f1c8684View commit details -
[fix][auto-recovery] Improve to the ReplicaitonWorker performance by …
…deleting invalid underreplication nodes (apache#21059)
Configuration menu - View commit details
-
Copy full SHA for ba0f2ba - Browse repository at this point
Copy the full SHA ba0f2baView commit details -
Configuration menu - View commit details
-
Copy full SHA for f35d3e0 - Browse repository at this point
Copy the full SHA f35d3e0View commit details -
Configuration menu - View commit details
-
Copy full SHA for eedbdb1 - Browse repository at this point
Copy the full SHA eedbdb1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7ecb93c - Browse repository at this point
Copy the full SHA 7ecb93cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 64d006b - Browse repository at this point
Copy the full SHA 64d006bView commit details -
[improve] [ml] Persist mark deleted ops to ZK if create cursor ledger…
… was failed (apache#20935) The progress Persist mark deleted position is like this: - persist to BK - If failed to persist to BK, try to persist to ZK But in the current implementation: if the cursor ledger was created failed, Pulsar will not try to persist to ZK. It makes if the cursor ledger created fails, a lot of ack records can not be persisted, and we will get a lot of repeat consumption after the BK recover. Modifications: Try to persist the mark deleted position to ZK if the cursor ledger was created failed
Configuration menu - View commit details
-
Copy full SHA for 843b830 - Browse repository at this point
Copy the full SHA 843b830View commit details
Commits on Sep 1, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 0956def - Browse repository at this point
Copy the full SHA 0956defView commit details -
Configuration menu - View commit details
-
Copy full SHA for 835e9b6 - Browse repository at this point
Copy the full SHA 835e9b6View commit details -
Configuration menu - View commit details
-
Copy full SHA for a25125d - Browse repository at this point
Copy the full SHA a25125dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 927d1b2 - Browse repository at this point
Copy the full SHA 927d1b2View commit details -
[fix][broker] Fix deleting topic not delete the related topic policy …
…and schema. (apache#21093) Fixes apache#21075 ### Motivation When the topic is loaded, it will delete the topic-level policy if it is enabled. But if the topic is not loaded, it will directly delete through managed ledger factory. But then we will leave the topic policy there. When the topic is created next time, it will use the old topic policy ### Modifications When deleting the topic, delete the schema and topic policies even if the topic is not loaded.
Configuration menu - View commit details
-
Copy full SHA for a1405ea - Browse repository at this point
Copy the full SHA a1405eaView commit details
Commits on Sep 3, 2023
-
Configuration menu - View commit details
-
Copy full SHA for cb24ab0 - Browse repository at this point
Copy the full SHA cb24ab0View commit details
Commits on Sep 4, 2023
-
[fix][client] Avoid ack hole for chunk message (apache#21101)
## Motivation Handle ack hole case: For example: ```markdown Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1 Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2 Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3 Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4 Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5 ``` Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the ChunkMessageIdImpl, so we should process it here. ## Modification Ack chunk-1 and chunk-2.
Configuration menu - View commit details
-
Copy full SHA for 59a8e72 - Browse repository at this point
Copy the full SHA 59a8e72View commit details -
[fix][broker] Avoid splitting one batch message into two entries in S…
…trategicTwoPhaseCompactor (apache#21091)
Configuration menu - View commit details
-
Copy full SHA for e59c850 - Browse repository at this point
Copy the full SHA e59c850View commit details -
[fix][broker] Fix write duplicate entries into the compacted ledger a…
…fter RawReader reconnects (apache#21081)
Configuration menu - View commit details
-
Copy full SHA for 2921a41 - Browse repository at this point
Copy the full SHA 2921a41View commit details -
[fix][broker] Cleanup correctly heartbeat bundle ownership when handl…
…ing broker deletion event (apache#21083)
Configuration menu - View commit details
-
Copy full SHA for b26ee8a - Browse repository at this point
Copy the full SHA b26ee8aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 55acb35 - Browse repository at this point
Copy the full SHA 55acb35View commit details