Destination S3 connector check ignores the s3_bucket_path

[itaiad200]

Environment

• Airbyte version: 0.35.65-alpha
• OS Version / Instance: macOS
• Deployment: docker-compose
• Source Connector and version: -
• Destination Connector and version: airbyte/destination-s3 0.3.0
• Severity: High
• Step where error happened: Setup new connection

Current Behavior

When setting up a new S3 destination, the worker tries to write an object with a key equal to the bucket while ignoring the given bucket path. For example, writing to bucket repo1 with bucket path foo/bar/, will try to write the object repo1/repo1. This might fail if the granted creds has limited access and can only use the bucket path, or for s3 compatible tools, where bucket path is a mandatory field (like a branch in lakeFS). I think this is the buggy line (But I suspect there are similar issues in the next lines as well :) )

Expected Behavior

The worker should respect the configured bucket path and use it as part of the check protocol.

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG

airbyte-worker      | 2022-04-10 22:38:19 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-10 22:38:19 INFO i.a.i.d.s.S3FormatConfigs(getS3FormatConfig):22 - S3 format config: {"compression_codec":{"codec":"no compression"},"part_size_mb":5,"format_type":"Avro"}
airbyte-worker      | 2022-04-10 22:38:19 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-10 22:38:19 INFO i.a.i.d.s.S3DestinationConfig(createS3Client):169 - Creating S3 client...
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-10 22:38:20 INFO i.a.i.d.s.S3StorageOperations(createBucketObjectIfNotExists):77 - Storage Object repo1/repo1 does not exist in bucket; creating...
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-04-10 22:38:20 ERROR i.a.i.d.s.S3Destination(check):72 - Exception attempting to access the S3 bucket:
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: null; S3 Extended Request ID: null; Proxy: null)
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530) ~[aws-java-sdk-core-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437) ~[aws-java-sdk-s3-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384) ~[aws-java-sdk-s3-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421) ~[aws-java-sdk-s3-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6508) ~[aws-java-sdk-s3-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1856) ~[aws-java-sdk-s3-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1816) ~[aws-java-sdk-s3-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:3982) ~[aws-java-sdk-s3-1.12.6.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.s3.S3StorageOperations.createBucketObjectIfNotExists(S3StorageOperations.java:78) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-0.35.64-alpha.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.s3.S3Destination.attemptWriteAndDeleteS3Object(S3Destination.java:147) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-0.35.64-alpha.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.s3.S3Destination.attemptS3WriteAndDelete(S3Destination.java:138) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-0.35.64-alpha.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.s3.S3Destination.attemptS3WriteAndDelete(S3Destination.java:128) ~[io.airbyte.airbyte-integrations.connectors-destination-s3-0.35.64-alpha.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.s3.S3Destination.check(S3Destination.java:62) [io.airbyte.airbyte-integrations.connectors-destination-s3-0.35.64-alpha.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:138) [io.airbyte.airbyte-integrations.bases-base-java-0.35.64-alpha.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:105) [io.airbyte.airbyte-integrations.bases-base-java-0.35.64-alpha.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 	at io.airbyte.integrations.destination.s3.S3Destination.main(S3Destination.java:51) [io.airbyte.airbyte-integrations.connectors-destination-s3-0.35.64-alpha.jar:?]
airbyte-worker      | 2022-04-10 22:38:20 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...

Steps to Reproduce

1. Setup any S3 destination with a bucket path. You may not fail if you're allowed to write to the entire bucket, so make sure to limit your credentials to that path.

Are you willing to submit a PR?

Remove this with your answer.

[haithem-souala]

Any updates, this is actually is blocking us from using Airbyte.

[N-o-Z]

Please note that after @haithem-souala's fix, this was broken again due to a refactoring PR