providers/aws: resource aws_s3_bucket keeps modifying policy

[pmoust]

Repro:

a. foobar.tf

resource "aws_s3_bucket" "foobar" {
    bucket  = "foobar666"
    acl     = "private"
    policy  = "${file("detailed_billing_s3.json")}"
    tags {
        Name = "Billing Reports"
    }
}

b. detailed_billing_s3.json

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Principal": {
      "AWS": "386209384616"
    },
    "Action": [
      "s3:GetBucketAcl",
      "s3:GetBucketPolicy"
    ],
    "Resource": "arn:aws:s3:::foobar666"
  },
  {
    "Effect": "Allow",
    "Principal": {
      "AWS": "386209384616"
    },
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::foobar666/*"
  }
  ]
}

Produced state:
state

{
    "version": 1,
    "serial": 3,
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {
                "aws_s3_bucket.foobar": {
                    "type": "aws_s3_bucket",
                    "primary": {
                        "id": "foobar666",
                        "attributes": {
                            "acl": "private",
                            "bucket": "foobar666",
                            "force_destroy": "false",
                            "hosted_zone_id": "Z3AQBSTGFYJSTF",
                            "id": "foobar666",
                            "policy": "{\"Statement\":[{\"Action\":[\"s3:GetBucketAcl\",\"s3:GetBucketPolicy\"],\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::386209384616:root\"},\"Resource\":\"arn:aws:s3:::foobar666\",\"Sid\":\"\"},{\"Action\":\"s3:PutObject\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::386209384616:root\"},\"Resource\":\"arn:aws:s3:::foobar666/*\",\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}",
                            "region": "us-east-1",
                            "tags.#": "1",
                            "tags.Name": "Billing Reports",
                            "website.#": "0",
                            "website_endpoint": ""
                        }
                    }
                }
            }
        }
    ]
}

Plan keeps giving:

~ aws_s3_bucket.foobar
    policy: "{\"Statement\":[{\"Action\":[\"s3:GetBucketAcl\",\"s3:GetBucketPolicy\"],\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::386209384616:root\"},\"Resource\":\"arn:aws:s3:::foobar666\",\"Sid\":\"\"},{\"Action\":\"s3:PutObject\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::386209384616:root\"},\"Resource\":\"arn:aws:s3:::foobar666/*\",\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}" => "{\"Statement\":[{\"Action\":[\"s3:GetBucketAcl\",\"s3:GetBucketPolicy\"],\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"386209384616\"},\"Resource\":\"arn:aws:s3:::foobar666\"},{\"Action\":\"s3:PutObject\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"386209384616\"},\"Resource\":\"arn:aws:s3:::foobar666/*\"}],\"Version\":\"2012-10-17\"}"

The actual policy on bucket is;

{
    "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::386209384616:root\"},\"Action\":[\"s3:GetBucketAcl\",\"s3:GetBucketPolicy\"],\"Resource\":\"arn:aws:s3:::foobar666\"},{\"Sid\":\"\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::386209384616:root\"},\"Action\":\"s3:PutObject\",\"Resource\":\"arn:aws:s3:::foobar666/*\"}]}"
}

Am I forced to be verbose about Principal and give its ARN instead of just account no.?

Should this be handled by the Create/Update func of aws_s3_bucket?

[pmoust]

Also seems that I have to enter Sid: ""

[pmoust]

@phinze solution was to use the following verbose policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::386209384616:root"
      },
      "Action": [
        "s3:GetBucketAcl",
        "s3:GetBucketPolicy"
      ],
      "Resource": "arn:aws:s3:::foobar666"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::386209384616:root"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::foobar666/*"
    }
  ]
}

I don't know if it's worth spending engineering time to fix this. It would certainly make UX better for us.

[mitchellh]

I'm not sure if its worth it either. It woudl require us to actually parse the policy which seems... unnecessary. I've updated the docs to note this though.

[coen-hyde]

@mitchellh Dealing with s3 bucket policy diffs is my largest time sink with Terraform. If a more intelligent way to deal with the policy diffs isn't practical. I suggest a trouble shooting section in the docs.

Things I've learnt:

• AWS will collapse arrays with one element. i.e. "Action": ["s3:PutObject"] becomes "Action": "s3:PutObject"
• AWS will change the ordering of elements in arrays arbitrarily (or at least i am not privy to their ordering scheme). i.e.

"Action": [
  "s3:ListBucket",
  "s3:ListBucketMultipartUploads",
  "s3:GetObject",
  "s3:PutObject"
],

Might become:

"Action": [
  "s3:ListBucket",
  "s3:GetObject",
  "s3:ListBucketMultipartUploads",
  "s3:PutObject"
],

• The "ordering" of array elements even differs between AWS accounts. One AWS account will reorganize a list one way and another AWS account will reorganize the same list another way. This is really frustrating when applying infrastructure to multiple accounts.

An example:
Account A modifies a resource list to be in this order:
"Resource": ["arn:aws:s3:::bucket/*", "arn:aws:s3:::bucket"]

Account B modifies a resource list to be in this order:
"Resource": ["arn:aws:s3:::bucket", "arn:aws:s3:::bucket/*"]

In this situation it is impossible to modify the Terraform s3 bucket policy to conform to what AWS produces. Ideally I think should Terraform do a deep equals with some smarts to resolve some of this.

[phinze]

@coen-hyde that is a really good idea. The docs are present in this repo under the website dir - do you think you could drop this section into the aws_s3_bucket resource page in a PR?

[rokka-n]

Ugh, silly AWS.

Copy/paste policy from the bucket itself into template.
Update any variables.
Save template.

No changes.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.