Skip to content
This repository has been archived by the owner on Apr 1, 2024. It is now read-only.

Bump pulsar version to 3.1.0-SNAPSHOT #5894

Closed
wants to merge 25 commits into from

Commits on Aug 29, 2023

  1. [fix][client] Fix consumer can't consume resent chunked messages (apa…

    …che#21070)
    
    ### Motivation
    
    Current, when the producer resend the chunked message like this:
    - M1: UUID: 0, ChunkID: 0
    - M2: UUID: 0, ChunkID: 0 // Resend the first chunk
    - M3: UUID: 0, ChunkID: 1
    
    When the consumer received the M2, it will find that it's already tracking the UUID:0 chunked messages, and will then discard the message M1 and M2. This will lead to unable to consume the whole chunked message even though it's already persisted in the Pulsar topic.
    
    Here is the code logic:
    https://github.com/apache/pulsar/blob/44a055b8a55078bcf93f4904991598541aa6c1ee/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1436-L1482
    
    The bug can be easily reproduced using the testcase `testResendChunkMessages` introduced by this PR.
    
    
    ### Modifications
    
    - When receiving the new duplicated first chunk of a chunked message, the consumer discard the current chunked message context and create a new context to track the following messages. For the case mentioned in Motivation, the M1 will be released and the consumer will assemble M2 and M3 as the chunked message.
    RobertIndie authored Aug 29, 2023
    Configuration menu
    Copy the full SHA
    eb2e3a2 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. Configuration menu
    Copy the full SHA
    eded9f1 View commit details
    Browse the repository at this point in the history
  2. [fix][build] Upgrade Guava to 32.1.2-jre (apache#21090)

    Masahiro Sakamoto authored Aug 30, 2023
    Configuration menu
    Copy the full SHA
    dab5b2f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    53ffe81 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    99e3fea View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2023

  1. [fix] [broker] remove bundle-data in local metadata store. (apache#21078

    )
    
    Motivation: When deleting a namespace, we will delete znode under the path `/loadbalance/bundle-data` in `local metadata store` instead of `global metadata store`.
    
    Modifications: Delete bundle data znode in local metadata store.
    thetumbled authored Aug 31, 2023
    Configuration menu
    Copy the full SHA
    4a87c64 View commit details
    Browse the repository at this point in the history
  2. [fix][broker]Fix chunked messages will be filtered by duplicating (ap…

    …ache#20948)
    
    ## Motivation
    Make the chunk message function work properly when deduplication is enabled.
    ## Modification
    ### Only check and store the sequence ID of the last chunk in a chunk message.
     For example:
     ```markdown
         Chunk-1 sequence ID: 0, chunk ID: 0, total chunk: 2
         Chunk-2 sequence ID: 0, chunk ID: 1
         Chunk-3 sequence ID: 1, chunk ID: 0 total chunk: 3
         Chunk-4 sequence ID: 1, chunk ID: 1
         Chunk-5 sequence ID: 1, chunk ID: 1
         Chunk-6 sequence ID: 1, chunk ID: 2
    ```   
    Only store check and store the sequence ID of Chunk-2 and Chunk-6.
    **Add a property in the publishContext to determine whether this chunk is the last chunk when persistent completely.**
    ```java
    publishContext.setProperty(IS_LAST_CHUNK, Boolean.FALSE);
    ```
    ### Filter and ack duplicated chunks in a chunk message instead of discarding ctx.
     For example:
     ```markdown
         Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
         Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
         Chunk-3 sequence ID: 0, chunk ID: 2, msgID: 1:3
         Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
         Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
         Chunk-6 sequence ID: 0, chunk ID: 3, msgID: 1:6
    ```   
    We should filter and ack chunk-4 and chunk-5.
    liangyepianzhou authored Aug 31, 2023
    Configuration menu
    Copy the full SHA
    b0b13bc View commit details
    Browse the repository at this point in the history
  3. [fix][io] Allow setting sourceType in config file (apache#19836)

    Signed-off-by: tison <[email protected]>
    Co-authored-by: Alexander Preuß <[email protected]>
    Co-authored-by: tison <[email protected]>
    3 people authored Aug 31, 2023
    Configuration menu
    Copy the full SHA
    f1c8684 View commit details
    Browse the repository at this point in the history
  4. [fix][auto-recovery] Improve to the ReplicaitonWorker performance by …

    …deleting invalid underreplication nodes (apache#21059)
    horizonzy authored Aug 31, 2023
    Configuration menu
    Copy the full SHA
    ba0f2ba View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f35d3e0 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    eedbdb1 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    7ecb93c View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    64d006b View commit details
    Browse the repository at this point in the history
  9. [improve] [ml] Persist mark deleted ops to ZK if create cursor ledger…

    … was failed (apache#20935)
    
    The progress Persist mark deleted position is like this:
    - persist to BK
    - If failed to persist to BK, try to persist to ZK
    
    But in the current implementation: if the cursor ledger was created failed, Pulsar will not try to persist to ZK. It makes if the cursor ledger created fails, a lot of ack records can not be persisted, and we will get a lot of repeat consumption after the BK recover.
    
    Modifications: Try to persist the mark deleted position to ZK if the cursor ledger was created failed
    poorbarcode authored Aug 31, 2023
    Configuration menu
    Copy the full SHA
    843b830 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. Configuration menu
    Copy the full SHA
    0956def View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    835e9b6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a25125d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    927d1b2 View commit details
    Browse the repository at this point in the history
  5. [fix][broker] Fix deleting topic not delete the related topic policy …

    …and schema. (apache#21093)
    
    Fixes apache#21075 
    
    ### Motivation
    
    When the topic is loaded, it will delete the topic-level policy if it is enabled. But if the topic is not loaded, it will directly delete through managed ledger factory. But then we will leave the topic policy there. When the topic is created next time, it will use the old topic policy
    
    ### Modifications
    
    When deleting the topic, delete the schema and topic policies even if the topic is not loaded.
    Technoboy- authored Sep 1, 2023
    Configuration menu
    Copy the full SHA
    a1405ea View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2023

  1. Configuration menu
    Copy the full SHA
    cb24ab0 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2023

  1. [fix][client] Avoid ack hole for chunk message (apache#21101)

    ## Motivation
    Handle ack hole case:
    For example:
    ```markdown
                         Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
                         Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
                         Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
                         Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
                         Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
    ```
     Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk
     message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
     ChunkMessageIdImpl, so we should process it here.
    ## Modification
    Ack chunk-1 and chunk-2.
    liangyepianzhou authored Sep 4, 2023
    Configuration menu
    Copy the full SHA
    59a8e72 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e59c850 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2921a41 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b26ee8a View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    55acb35 View commit details
    Browse the repository at this point in the history