[fix][broker] upgrade jclouds 2.5.0 -> 2.6.0

[pgier]

Upgrade jclouds to fix offload issue on GCP clusters.

Fixes #15159

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• [ X ] doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: pgier#11

[github-actions]

@pgier Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[pgier]

Hmm, I guess this PR was premature, 2.6.0 has only released an RC but not the final release yet. I'll leave this open for when they do release 2.6.0.

[dave2wave]

Here's the vote thread: https://lists.apache.org/thread/47zpc6sqjhmwwsrvtl1kz5mmjjxk12hm

It looks like once Andrew votes it will pass and the release will happen.

[lhotari]

LGTM

[lhotari]

@dao-jun it seems that there are failures after upgrading to 2.6.0 and closing and reopening to get the latest master changes won't make a difference.

[eolivelli]

It seems that there is some work to do, I am seeing these errors on CI:

BlobStoreBackedInputStreamTest.start
Unable to create injector, see the following errors:

1) [Guice/MissingImplementation]: No implementation for inject.Provider<Gson> was bound.

Did you mean?
    Gson bound at GsonModule.provideGson(GsonModule.java:103)
      \_ installed by: RestModule -> GsonModule

Requested by:
1  : GsonModule$PropertiesAdapter.<init>(GsonModule.java:260)
      \_ for 1st parameter
     at GsonModule.provideGson(GsonModule.java:103)
      \_ for 5th parameter
     at GsonModule.provideGson(GsonModule.java:103)
      \_ installed by: RestModule -> GsonModule

[lhotari]

@pgier do you have a chance to check the test failures and fix them? thanks

[pgier]

Looks like jclouds upgrade requires a gson upgrade.
Edit: I tried updating the global gson, but it seems that this causes other failures.

[dao-jun]

Looks like jclouds upgrade requires a gson upgrade.

Please help resolve these failures,

[dao-jun]

Since I upgraded guice to 7.0.0, seems still cannot fix these failures

[lhotari]

Since I upgraded guice to 7.0.0, seems still cannot fix these failures

@dao-jun It should be possible to set the versions specificly to jclouds-shaded/pom.xml, in a dependencyManager block so that the versions are only applied to jclouds-shaded within Pulsar.
Example of this approach:

pulsar/pulsar-io/alluxio/pom.xml

Lines 86 to 108 in 37d88b8

	<dependencyManagement>
	<dependencies>
	<dependency>
	<groupId>io.netty</groupId>
	<artifactId>netty-bom</artifactId>
	<version>${netty.version}</version>
	<type>pom</type>
	<scope>import</scope>
	</dependency>
	<dependency>
	<groupId>io.grpc</groupId>
	<artifactId>grpc-bom</artifactId>
	<version>${grpc.version}</version>
	<type>pom</type>
	<scope>import</scope>
	</dependency>
	<dependency>
	<groupId>io.dropwizard.metrics</groupId>
	<artifactId>metrics-jvm</artifactId>
	<version>${metrics.version}</version>
	</dependency>
	</dependencies>
	</dependencyManagement>

.

[lhotari]

@pgier @dave2wave This has been merged to master branch now. do you have a chance to run some tests with the changes to see if it really fixes the issues with >2GB on GCP?

[lhotari]

@pgier @dave2wave This has been merged to master branch. do you have a chance to run some tests with the changes to see if it really fixes the issues with >2GB on GCP?

[pgier]

@lhotari Thanks for fixing this. We should be able to test it, but maybe not for a couple more weeks.

[lhotari]

@lhotari Thanks for fixing this. We should be able to test it, but maybe not for a couple more weeks.

@pgier do you have any test results to share about JClouds 2.6.0 ? I'm thinking of cherry-picking to maintenance branches after there's more confirmation that JClouds 2.6.0 works well with Pulsar. /cc @dave2wave @eolivelli @nicoloboschi

[nikhil-ctds]

@lhotari With the above fix I tried testing GCS offloader with ledgers of size >= 2.5gb. I'm still facing the same error

Error in offload
null

Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/bucket-cognitree-ls1258/o/d7f69abd-dc45-4fc1-bbe3-bd9daaa8fea7-ledger-9/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
  "error": {
    "code": 400,
    "message": "The number of source components provided (38) exceeds the maximum (32)",
    "errors": [
      {
        "message": "The number of source components provided (38) exceeds the maximum (32)",
        "domain": "global",
        "reason": "invalid"
      }
    ]
  }
}
]

I have also written a simple code using Jclouds 2.6.0 to upload a large file of size 3gb to GCS bucket, it has uploaded successfully.

[lhotari]

@lhotari With the above fix I tried testing GCS offloader with ledgers of size >= 2.5gb. I'm still facing the same error

Error in offload
null

Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/bucket-cognitree-ls1258/o/d7f69abd-dc45-4fc1-bbe3-bd9daaa8fea7-ledger-9/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
  "error": {
    "code": 400,
    "message": "The number of source components provided (38) exceeds the maximum (32)",
    "errors": [
      {
        "message": "The number of source components provided (38) exceeds the maximum (32)",
        "domain": "global",
        "reason": "invalid"
      }
    ]
  }
}
]

I have also written a simple code using Jclouds 2.6.0 to upload a large file of size 3gb to GCS bucket, it has uploaded successfully.

@nikhil-ctds Do you have a chance to rerun the test with gcsManagedLedgerOffloadMaxBlockSizeInBytes=134217728 ?
The default value gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864 will limit the max upload size to 2048MiB (64MiB * 32). Doubling gcsManagedLedgerOffloadMaxBlockSizeInBytes to 134217728 should lift the limit to 4096MiB.
Can you please verify? Thanks

[lhotari]

@nikhil-ctds I created #22554 that changes the default value of gcsManagedLedgerOffloadMaxBlockSizeInBytes. It would be helpful if you could test that gcsManagedLedgerOffloadMaxBlockSizeInBytes=134217728 fixes the issue. thanks

[nikhil-ctds]

@lhotari I have tested with gcsManagedLedgerOffloadMaxBlockSizeInBytes=134217728, and was able to offload ledger of 3.5gb to gcs bucket.