provider/aws: Weird behavior around ignore_changes

[Bowbaq]

Use case: ignore changes to the shard count on a kinesis stream

TL/DR

• there appears to be some weird interactions between ignore_changes and fields marked as ForceNew: true
• changing the shard_count on a kinesis stream managed by terraform causes that stream to be replaced by a new one, deleting all the data
• replacing a resource doesn't always cause its ARN to change. It would be nice if a change that causes a resource to be replaced but retain its ARN didn't trickle down to all dependent resources

Problem 1: can't create a stream while ignore changes to shard_count

resource "aws_kinesis_stream" "test" {
  name = "tf-test-stream"
  shard_count = 1

  lifecycle {
    ignore_changes = [ "shard_count" ]
  }
}

Expected behavior: kinesis stream gets created, subsequent changes to shard_count are ignored
Actual:

* aws_kinesis_stream.test: [WARN] Error creating Kinesis Stream: "1 validation error detected: Value '0' at 'shardCount' failed to satisfy constraint: Member must have value greater than or equal to 1", code: "ValidationException"

Problem 2: external changes to the shard count cause terraform to re-create the stream

• create stream without the ignore_changes modifier

resource "aws_kinesis_stream" "test" {
  name = "tf-test-stream"
  shard_count = 1
}

bowbaq@bowbaq-originate [03:53:16] [/tmp/tf] 
-> % terraform apply
aws_kinesis_stream.test: Creating...
  arn:              "" => "<computed>"
  name:             "" => "tf-test-stream"
  retention_period: "" => "24"
  shard_count:      "" => "1"
aws_kinesis_stream.test: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

• add the ignore_changes modifier
• manually re-shard the stream
  Expected behavior: nothing happens
  Actual: terraform plan wants to re-create the stream

bowbaq@bowbaq-originate [03:56:40] [/tmp/tf] 
-> % terraform plan
Refreshing Terraform state prior to plan...

aws_kinesis_stream.test: Refreshing state... (ID: arn:aws:kinesis:us-east-1:314531994667:stream/tf-test-stream)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

-/+ aws_kinesis_stream.test
    arn:              "arn:aws:kinesis:us-east-1:314531994667:stream/tf-test-stream" => "<computed>"
    name:             "tf-test-stream" => "tf-test-stream"
    retention_period: "24" => "24"


Plan: 1 to add, 0 to change, 1 to destroy.

Problem 3: number of shards read by terraform includes closed shards

Expected: terraform only considers open shards in the shard count. closed shards expire after a while.
Actual: number of open shards: 2, number of closed shards: 1

-/+ aws_kinesis_stream.test
    arn:              "arn:aws:kinesis:us-east-1:314531994667:stream/tf-test-stream" => "<computed>"
    name:             "tf-test-stream" => "tf-test-stream"
    retention_period: "24" => "24"
    shard_count:      "3" => "1" (forces new resource)


Plan: 1 to add, 0 to change, 1 to destroy.

Problem 4: resource replacement causes dependents to be updated, even if ARN remains the same

resource "aws_kinesis_stream" "test" {
  name = "tf-test-stream"
  shard_count = 1

  lifecycle {
    ignore_changes = [ "shard_count" ]
  }
}

resource "aws_iam_policy" "test" {
  name = "tf-test-stream-read"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:GetRecords",
        "kinesis:GetShardIterator",
        "kinesis:DescribeStream",
        "kinesis:ListStreams"
      ],
      "Resource": "${aws_kinesis_stream.test.arn}"
    }
  ]
}
EOF
}

bowbaq@bowbaq-originate [04:34:51] [/tmp/tf] 
-> % terraform plan
Refreshing Terraform state prior to plan...

aws_kinesis_stream.test: Refreshing state... (ID: arn:aws:kinesis:us-east-1:314531994667:stream/tf-test-stream)
aws_iam_policy.test: Refreshing state... (ID: arn:aws:iam::314531994667:policy/tf-test-stream-read)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

~ aws_iam_policy.test
    policy: "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"kinesis:GetRecords\",\n        \"kinesis:GetShardIterator\",\n        \"kinesis:DescribeStream\",\n        \"kinesis:ListStreams\"\n      ],\n      \"Resource\": \"arn:aws:kinesis:us-east-1:314531994667:stream/tf-test-stream\"\n    }\n  ]\n}\n" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"kinesis:GetRecords\",\n        \"kinesis:GetShardIterator\",\n        \"kinesis:DescribeStream\",\n        \"kinesis:ListStreams\"\n      ],\n      \"Resource\": \"${aws_kinesis_stream.test.arn}\"\n    }\n  ]\n}\n"

-/+ aws_kinesis_stream.test
    arn:              "arn:aws:kinesis:us-east-1:314531994667:stream/tf-test-stream" => "<computed>"
    name:             "tf-test-stream" => "tf-test-stream"
    retention_period: "24" => "24"


Plan: 1 to add, 1 to change, 1 to destroy.

[joshuaspence]

+1

[Bowbaq]

@phinze @catsby do you guys have any ideas about this?

[phinze]

Hi @Bowbaq - thanks for this report - I'm diving into ignore_changes issues today. Consolidating these problems up into: #5627

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.