[improve][broker, build, client, fn, io, ml, proxy, sql, test] Merge apache/branch-3.1 into 3.1_ds #212
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikhil-ctds
requested review from
eolivelli,
nicoloboschi,
NadirJ and
zzzming
nicoloboschi
approved these changes
Dec 20, 2023
(cherry picked from commit 1bf7371)
apache#21278) (cherry picked from commit 682eb36) # Conflicts: # pom.xml
…ear (apache#21302) (cherry picked from commit 70d086f)
…AuthorizationProvider plugin (apache#20939) Fixed apache#20938
…apache#21338) Co-authored-by: Lari Hotari <[email protected]>
…g from BookKeeper to Pulsar. (apache#21340) There is no testing for AutoRecovery replication in Pulsar's current test suite, and we need to cover it. So migrate the replication testing from BookKeeper to Pulsar.
Fix zookeeper related flacky test (cherry picked from commit 9ab7417)
…re building (apache#21365) (cherry picked from commit 421c98a)
apache#21155) The client assumed the connection was inactive, but the Broker assumed the connection was fine. The Client tried to use a new connection to reconnect a producer, then got an error `Producer with name 'st-0-5' is already connected to topic`. - In a connection, the second connection waits for the first connection to complete\. But there is a bug that causes this mechanism to fail\. - If a producer uses a default name, the second registration will override the first one. But it can not override the first one if it uses a specified producer name\. I think this mechanism is to prevent a client from creating two producers with the same name. However, method `Producer.isSuccessorTo` has checked the `producer-id`, and the `producer-id` of multiple producers created by the same client are different. So this mechanism can be deleted. - For `issue 1`: If a producer with the same name tries to use a new connection, async checks the old connection is available. The producers related to the connection that is not available are automatically cleaned up. - For `issue 2`: - Fix the bug that causes a complete producer future will be removed from `ServerCnx`. - Remove the mechanism that prevents a producer with a specified name from overriding the previous producer. (cherry picked from commit bda16b6)
(cherry picked from commit 39235ed) # Conflicts: # pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ClusterMigrationTest.java
### Motivation As https://pulsar.apache.org/docs/3.1.x/sql-overview/, Pulsar SQL is based on [Trino (formerly Presto SQL)](https://trino.io/), which supports UUID type. But now, the UUID field in Avro or JSON schema will be interpreted as VARCHAR. ### Modifications Support decoding UUID form AVRO or JSON schema. (cherry picked from commit 8c70943)
(cherry picked from commit e6cd11f)
) (cherry picked from commit ffc083b)
…le when using extensible load manager (apache#21213) (apache#21314)
…r migration tests (apache#21391) (cherry picked from commit e2c6c08)
…o after bkclient initialize (apache#20944)
…elete heartbeat topic (apache#21360) Co-authored-by: fanjianye <[email protected]> Co-authored-by: Jiwei Guo <[email protected]>
…MinNumRacksPerWriteQuorum not work problem. (apache#21327)
…d avoid recursive update error (apache#21282) (cherry picked from commit aecdb03)
…er (apache#21332) (cherry picked from commit ecd40e4)
…21619) Co-authored-by: Jiwe Guo <[email protected]>
…he#21447) ### Motivation Since the files will be checked for `LICENSE` whose name is a suffix with `.conf`. But we can not add a LICENSE definition for the file `schema_example.conf` because it is a `JSON` file. Note: the file `schema_example.conf` is only used for the integration test `CLITest.java`. ### Modifications Rename `schema_example.conf` to `schema_example.json`.
… if the previous one is inactive (apache#21220) If a producer establishes a new connection when it is reconnecting, while the previous connection is now inactive, the initial request made on the new connection will fail. This failure will trigger the topic of cleaning up the inactive producers. However, upon making a second request, the producer will be able to successfully establish a connection and proceed with the operation. Make the initial request made on the new connection success. (cherry picked from commit 711b621)
### Motivation
When I did this test below and got the error "Topic not found".
```java
pulsarAdmin.topics().createNonPartitionedTopic("persistent://public/default/tp_1");
Consumer consumer = null;
Consumer consumer = pulsarClient.newConsumer()
.topic("persistent://public/default/tp_1")
.subscriptionName("s1")
.enableRetry(true)
.subscribe();
```
I do create the topic `persistent://public/default/tp_1` first but got a response "Topic not found", it is confusing.
The root cause is the retry letter topic `persistent://public/default/tp_1-sub1-RETRY` was not created.
### Modifications
clear the vague response if the topic is not founded.
(cherry picked from commit 1a024bc)
…n on non-persistent topic (apache#21592) Fixes apache#21568 Motivation Fix incorrect unack count when using shared subscription on non-persistent topic Modifications In the case of a non-persistent topic, the consumer does not send an ack to the broker (see org.apache.pulsar.client.impl.NonPersistentAcknowledgmentGroupingTracker# addAcknowledgment) To work around this, we can update unackedMessages when the broker sends a message to the consumer successfully.
Co-authored-by: Lari Hotari <[email protected]> Co-authored-by: xiangying <[email protected]> Co-authored-by: Jiwe Guo <[email protected]>
…endAsync (apache#21673) Signed-off-by: Zixuan Liu <[email protected]>
Signed-off-by: tison <[email protected]>
…ing when spy is stubbed
mvn initialize fails to discover all the sub-modules if the release versions are not available in mvn repo. This would fail only incase of new releases where the artifacts are not yet published into the mvn repo. Error: Failed to execute goal on project pulsar-client-all-shade-test: Could not resolve dependencies for project org.apache.pulsar.tests:pulsar-client-all-shade-test:jar:3.1.2: Failed to collect dependencies at com.datastax.oss:pulsar-client-all:jar:3.1.2: Failed to read artifact descriptor for com.datastax.oss:pulsar-client-all:jar:3.1.2: Could not transfer artifact com.datastax.oss:pulsar-client-all:pom:3.1.2 from/to
…pache#21684) Fixes apache#21557 ### Motivation There is a network package loss issue after enabling `haProxyProtocolEnabled`, which leads the error `Checksum failed on the broker` and `Adjusted frame length exceeds`, you can reproduce the issue by the test `testSlowNetwork`. ### Modifications Fix the bug. (cherry picked from commit 6e18874)
nikhil-ctds
force-pushed
the
3.1_ds_merge-branch-3.1
branch
from
908f611
to
96d9a32
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
This PR merges apache/branch-3.1 into 3.1_ds.
3.1_ds is currently on par with apache/v3.1.1, this merge makes the branch on par with apache/branch-3.1 which is at release v3.1.2
Modifications
Describe the modifications you've done.
Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?
doc-required(If you need help on updating docs, create a doc issue)
no-need-doc(Please explain why)
doc(If this PR contains doc changes)