This repository has been archived by the owner on Apr 1, 2024. It is now read-only.
Bump pulsar version to 3.1.0-SNAPSHOT #5898
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r. (apache#20880) Main Issue: apache#20851 ### Motivation When the Proto version does not allow us to send TcClientConnectRequest to the broker, we should add a log to debug it. ### Modifications Add a waining log.
…edup and set namespace disable deduplication. (apache#20905)
(cherry picked from commit 3ab420c)
(cherry picked from commit 3ab420c)
Fixes: apache#20997 ### Motivation Update the expired certs to get tests passing. ### Modifications * Update all certs. See `README.md` in files for detailed steps. ### Verifying this change This change is covered by tests. ### Documentation - [x] `doc-not-needed` (cherry picked from commit d6734b7)
…Horizon has been updated (apache#20984) (cherry picked from commit 0cb1c78)
…currently (apache#20971) ### Motivation **Background**: when calling `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>`, Pulsar will read the first entry which is not acknowledged, and respond with the entry write time. The flow is like this: - get the mark deleted position of the subscription - if no backlog, response `-1` - else read the next position of the mark deleted position, and respond with the entry write time. **Issue**: if the command `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>` and `consumer.acknowledge` are executed at the same time, the step 2 in above flow will get a position which is larger than the last confirmed position, lead a read entry error. | time | `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>` | `consumer.acknowledge` | | --- | --- | --- | | 1 | mark deleted position is `3:1` and LAC is `3:2` now | | 2 | the check `whether has backlog` is passed | | 3 | | acknowledged `3:2`, mark deleted position is `3:2` now | | 4 | calculate next position: `3:3` | | 5 | Read `3:3` and get an error: `read entry failed` | Note: the test in PR is not intended to reproduce the issue. ### Modifications Respond `-1` if the next position of the mark deleted position is larger than the LAC
…data. sec ver. (apache#20620) Co-authored-by: wangjinlong <[email protected]>
(cherry picked from commit d06cda6)
…ageId read reaches lastReadId (apache#20988) (cherry picked from commit 9e2195c)
…letion of compaction (apache#21067) (cherry picked from commit bb9c9b4)
…che#21070) ### Motivation Current, when the producer resend the chunked message like this: - M1: UUID: 0, ChunkID: 0 - M2: UUID: 0, ChunkID: 0 // Resend the first chunk - M3: UUID: 0, ChunkID: 1 When the consumer received the M2, it will find that it's already tracking the UUID:0 chunked messages, and will then discard the message M1 and M2. This will lead to unable to consume the whole chunked message even though it's already persisted in the Pulsar topic. Here is the code logic: https://github.com/apache/pulsar/blob/44a055b8a55078bcf93f4904991598541aa6c1ee/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1436-L1482 The bug can be easily reproduced using the testcase `testResendChunkMessages` introduced by this PR. ### Modifications - When receiving the new duplicated first chunk of a chunked message, the consumer discard the current chunked message context and create a new context to track the following messages. For the case mentioned in Motivation, the M1 will be released and the consumer will assemble M2 and M3 as the chunked message. (cherry picked from commit eb2e3a2)
… was failed (apache#20935) The progress Persist mark deleted position is like this: - persist to BK - If failed to persist to BK, try to persist to ZK But in the current implementation: if the cursor ledger was created failed, Pulsar will not try to persist to ZK. It makes if the cursor ledger created fails, a lot of ack records can not be persisted, and we will get a lot of repeat consumption after the BK recover. Modifications: Try to persist the mark deleted position to ZK if the cursor ledger was created failed
…ache#20948) ## Motivation Make the chunk message function work properly when deduplication is enabled. ## Modification ### Only check and store the sequence ID of the last chunk in a chunk message. For example: ```markdown Chunk-1 sequence ID: 0, chunk ID: 0, total chunk: 2 Chunk-2 sequence ID: 0, chunk ID: 1 Chunk-3 sequence ID: 1, chunk ID: 0 total chunk: 3 Chunk-4 sequence ID: 1, chunk ID: 1 Chunk-5 sequence ID: 1, chunk ID: 1 Chunk-6 sequence ID: 1, chunk ID: 2 ``` Only store check and store the sequence ID of Chunk-2 and Chunk-6. **Add a property in the publishContext to determine whether this chunk is the last chunk when persistent completely.** ```java publishContext.setProperty(IS_LAST_CHUNK, Boolean.FALSE); ``` ### Filter and ack duplicated chunks in a chunk message instead of discarding ctx. For example: ```markdown Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1 Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2 Chunk-3 sequence ID: 0, chunk ID: 2, msgID: 1:3 Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4 Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5 Chunk-6 sequence ID: 0, chunk ID: 3, msgID: 1:6 ``` We should filter and ack chunk-4 and chunk-5.
…fter RawReader reconnects (apache#21081)
…ing broker deletion event (apache#21083)
…trategicTwoPhaseCompactor (apache#21091)
## Motivation
Handle ack hole case:
For example:
```markdown
Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
```
Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk
message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
ChunkMessageIdImpl, so we should process it here.
## Modification
Ack chunk-1 and chunk-2.
…deleting invalid underreplication nodes (apache#21059)
…pache#20952) ### Motivation Minor improve `getIsolationGroup` by avoid creating arrayList. And it' beneficial to GC ### Modifications `new HashSet<>(Arrays.asList(...))` -> `Sets.newHashSet(...)`
### Modifications When upgraded the pulsar version from 2.9.2 to 2.10.3, and the isolated group feature not work anymore. Finally, we found the problem. In IsolatedBookieEnsemblePlacementPolicy, when it gets the bookie rack from the metadata store cache, uses future.isDone() to avoid sync operation. If the future is incomplete, return empty blacklists. The cache may expire due to the caffeine cache `getExpireAfterWriteMillis` config, if the cache expires, the future may be incomplete. (apache#21095 will correct the behavior) In 2.9.2, it uses the sync to get data from the metadata store, we should also keep the behavior.
…and schema. (apache#21093) Fixes apache#21075 ### Motivation When the topic is loaded, it will delete the topic-level policy if it is enabled. But if the topic is not loaded, it will directly delete through managed ledger factory. But then we will leave the topic policy there. When the topic is created next time, it will use the old topic policy ### Modifications When deleting the topic, delete the schema and topic policies even if the topic is not loaded.
…eystore TLS and admin API is called (apache#21077)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a PR created by snbot to trigger the check suite in each repository.