Bug in S3Client?

[stefanofornari]

Hi, sorry to spam this repository, but can you point me where I can file the following bug.

Steps to reproduce

1. set up a fake listener (e,g, nc -l 8880)
2. Make a list objects call:

S3Client s3Client = S3Client.builder()
    .endpointOverride(URI.create("http://localhost:8880"))
    .region(Region.EU_CENTRAL_1)
    .credentialsProvider(() -> AwsBasicCredentials.create("key", "secret"))
    .build();
    s3Client.listObjectsV2(
        ListObjectsV2Request.builder().bucket("pippo").build()
    );

Expected behaviour
A request to /pippo is issued, like:

GET /pippo?list-type=2 HTTP/1.1
Host: localhost:8880
amz-sdk-invocation-id: cef3a91f-bf57-cfbb-1fb8-45abea064b71
amz-sdk-request: attempt=1; max=4
Authorization: AWS4-HMAC-SHA256 Credential=key/20230715/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=ce93b4debefcf8f94fc57a6107584b06cbe712fe4d8228f77a69599b04997be2
User-Agent: aws-sdk-java/2.20.79 Linux/5.4.0-150-generic OpenJDK_64-Bit_Server_VM/19.0.1+10 Java/19.0.1 vendor/Azul_Systems__Inc. io/sync http/Apache cfg/retry-mode/legacy ft/s3-transfer
x-amz-content-sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20230715T093511Z
Connection: Keep-Alive

Current result

GET /?list-type=2 HTTP/1.1
Host: pippo.localhost:8880
amz-sdk-invocation-id: e6dd0685-09a1-807e-c9a0-7ead5f096fcb
amz-sdk-request: attempt=1; max=4
Authorization: AWS4-HMAC-SHA256 Credential=key/20230715/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=543f22241699c0337c2a17b9d07c84620255a371098dc636ac78ff0f351a18bf
User-Agent: aws-sdk-java/2.20.79 Linux/5.4.0-150-generic OpenJDK_64-Bit_Server_VM/19.0.1+10 Java/19.0.1 vendor/Azul_Systems__Inc. io/sync http/Apache cfg/retry-mode/legacy ft/s3-transfer
x-amz-content-sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20230715T092801Z
Connection: Keep-Alive

Notes

1. As you can see the bucket name goes in the Host header instead of on the GET line
2. It works correctly using the IP address in place of localhost in the given endpoint (e.g. http://127.0.0.1:8880)

[dannycjones]

Hey @stefanofornari. The AWS Java SDK is working as expected - by default it'll be using 'virtual-style addressing'.

This means is that the bucket name makes up part of the hostname. As an example, bucket pippo would route to pippo.s3.eu-central-1.amazonaws.com.

When working with third-party object stores (such as one on localhost), you may want to force the S3 client to use 'path-style addressing' if the store does not support virtual-style.

You can set this for the S3 client using this option: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3BaseClientBuilder.html#forcePathStyle(java.lang.Boolean)

[stefanofornari]

Hi @dannycjones,
thanks for the explanation! it is clear now.
@markjschreiber do you see any implication for the provider in general?
WRT 3rd party services, I guess we need to find a way to pass this configuration to the provider/file system

[markjschreiber]

do you see any implication for the provider in general?

Certainly some mechanism to allow the customization of clients will be required for 3rd party S3 like services but would also benefit clients for AWS S3 where a user wants to use something other than the defaults.

Possibly the FileSystemProvider could be composable and use a map of properties to values to override defaults. I think the FileSystemProvider is the logical entity that will be responsible for configuring and building clients.

[stefanofornari]

This is a request coming from the field as well (see stefanofornari#5); wecan address it as suggested when we split the provider to support 3rd party services by the means of a dedicated URI scheme.

[stefanofornari]

Implemented in v1.2.4