Unable to create S3 buckets on local mocks since 3.0.0

[ballwood]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Affected Resource(s)

• aws_s3_bucket

Terraform Configuration Files

provider "aws" {
  region = "eu-west-1"

  access_key = "fakeKey"
  secret_key = "fakeKey"

  skip_credentials_validation = true
  skip_metadata_api_check = true
  s3_force_path_style = true
  skip_requesting_account_id = true

  endpoints {
    s3 = "http://localhost:9090"
  }
}

resource "aws_s3_bucket" "test" {
  bucket = "test-bucket"
}

Debug Output

Panic Output

Expected Behavior

Bucket should have been created on s3 mock

Actual Behavior

Error: error getting S3 Bucket location: RequestError: send request failed caused by: dial tcp: lookup test-bucket.localhost on 192.168.1.1:53: no such host

Steps to Reproduce

1. docker run -p 9090:9090 -t adobe/s3mock
2. terraform apply

Important Factoids

• Started failing in 3.0 after new GetBucketLocation code was merged (Amazon S3 buckets region lookup with FIPS endpoints will fail with dial tcp errors. #14217)
• I would assume it has to do with aws.Bool(false) being set to false here rather than respecting s3_force_path_style = true in the provider

References

• Amazon S3 buckets region lookup with FIPS endpoints will fail with dial tcp errors. #14217

[bflad]

Hi @ballwood 👋 Thank you for reporting this and sorry for the trouble. At the moment we don't regularly test against non-AWS implementations, so apologies this was missed, but we certainly would like to provide best effort support for this functionality. Your suggestion about threading through the provider s3_force_path_style configuration seems to make the most sense. Would you like to submit the fix?

[artamonovkirill]

Here's an example of a test with Localstack: source, CI, Gradle Build.

I'd like to convert it to a go test on s3 resource, but:

• my go knowledge is rather limited, so it might take a while and might not be of the best quality.
• I'm not sure where it belongs to (since it required Docker) - tests? acceptance test?

[bflad]

@artamonovkirill you are not missing anything at the moment. 😅 Our acceptance testing framework expects that endpoints are running before testing begins and that configurations can be provided via environment variables, so we do not typically run our acceptance test against non-production AWS endpoints, such as Localstack, s3mock, etc. at the moment. We still list non-AWS implementations as best effort support due to maintainer team size and I'm not sure we have the bandwidth to change that position for now, even if submissions were made to enable that type of testing via Makefile targets and customized test configurations, etc.

Since we would like to fix this regression shortly though, please note that the fix has been submitted to passthrough the provider s3_force_path_style configuration (defaulting to false if not set) instead of hardcoding the value ( #14481) which was verified locally using the provided reproduction configuration and steps.

[bflad]

The fix for this (passing through the provider s3_force_path_style configuration) has been merged and will release with version 3.1.0 of the Terraform AWS Provider, likely later today. 👍

[artamonovkirill]

@bflad , thanks! 👍

[ballwood]

@bflad thanks for sorting this so promptly, appreciate it

[ghost]

This has been released in version 3.1.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[artamonovkirill]

@bflad , I'm still getting an error with 3.1.0 - a different one, though:
https://scans.gradle.com/s/5qmfxlaedn32c/tests/:test/AwsProviderSpec/creates%20an%20S3%20bucket%20with%203.1.0%20AWS%20provider#1

Are there other occurrences in the code where s3_force_path_style flag is ignored?

[bflad]

@artamonovkirill I think filing a separate bug report with the relevant details (e.g. what service/tooling you are using) is the most appropriate action in that case. Please do note here that the maintainers cannot guarantee full compatibility with all non-AWS tooling if they do not implement the S3 APIs correctly.

[artamonovkirill]

@bflad, thanks for the support.
I've double-checked, and the problem is indeed on the LocalStack side (localstack/localstack#2832, not related to API implementation per se though).

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!