Changes to resource list count unnecessarily destroys referencing resources using splat

[justinclayton]

Unfortunately #2788 seems to have introduced some unexpected behavior. Some of it has already been described in #3885, #3449, and elsewhere, but I wanted to provide a very simple and clear example of what I believe to be the culprit at the heart of these issues:

# test.tf
resource "template_file" "foo" {
  template = "I AM ${count.index}"
  count = 2
}

resource "template_file" "no_splat" {
  template = "the first foo says ${template_file.foo.0.rendered}"
}

resource "template_file" "with_splat" {
  template = "the first foo says ${element(template_file.foo.*.rendered, 0)}"
}

With both template_file.no_splat and template_file.with_splat we should expect the same result. And indeed we do, on the first plan and apply:

$ terraform plan
Refreshing Terraform state prior to plan...

<...>

+ template_file.foo.0
    rendered: "" => "<computed>"
    template: "" => "I AM 0"

+ template_file.foo.1
    rendered: "" => "<computed>"
    template: "" => "I AM 1"

+ template_file.no_splat
    rendered: "" => "<computed>"
    template: "" => "the first foo says ${template_file.foo.0.rendered}"

+ template_file.with_splat
    rendered: "" => "<computed>"
    template: "" => "the first foo says ${element(template_file.foo.*.rendered, 0)}"


Plan: 4 to add, 0 to change, 0 to destroy.

$ terraform apply
template_file.foo.1: Creating...
  rendered: "" => "<computed>"
  template: "" => "I AM 1"
template_file.foo.0: Creating...
  rendered: "" => "<computed>"
  template: "" => "I AM 0"
template_file.foo.0: Creation complete
template_file.foo.1: Creation complete
template_file.no_splat: Creating...
  rendered: "" => "<computed>"
  template: "" => "the first foo says I AM 0"
template_file.with_splat: Creating...
  rendered: "" => "<computed>"
  template: "" => "the first foo says I AM 0"
template_file.with_splat: Creation complete
template_file.no_splat: Creation complete

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Good so far. Now we increment template_file.foo.count from 2 to 3. Our other resources still only ever reference template_file.foo.0, and not this new resource being created. We run plan again:

$ terraform plan
Refreshing Terraform state prior to plan...

template_file.foo.0: Refreshing state... (ID: aabd55fb4b7b23363c246265d1c4daedc3798149a7230f1c082c8994aa0db39c)
template_file.foo.1: Refreshing state... (ID: 2b9e553f514d0789ecbee4ec00710b11645aebe744bd3de99b1ab6af9130c1e8)
template_file.with_splat: Refreshing state... (ID: 7f2cbd10b2517e514a9a7a6052f08a17e16b829df003f60e612b56f533e982dd)
template_file.no_splat: Refreshing state... (ID: 7f2cbd10b2517e514a9a7a6052f08a17e16b829df003f60e612b56f533e982dd)

<...>

+ template_file.foo.2
    rendered: "" => "<computed>"
    template: "" => "I AM 2"

-/+ template_file.with_splat
    rendered: "the first foo says I AM 0" => "<computed>"
    template: "the first foo says I AM 0" => "the first foo says ${element(template_file.foo.*.rendered, 0)}" (forces new resource)


Plan: 2 to add, 0 to change, 1 to destroy.

Here we can see:

• template_file.no_splat registers no change (because it references template_file.foo.0.rendered directly)
• template_file.with_splat is marked to be destroyed and recreated

This is even though the value of template_file.foo.0.rendered was never altered as part of this change. Harmless in this case, but needlessly destructive in a host of other common real-world scenarios, like when instances are rebuilt because of userdata (#3449), or when LB pool members must be destroyed and recreated before adding additional ones (#4359).

I've tested this same scenario after rolling back the changes made in #2788, and this time I get the expected result from plan:

$ terraform plan
Refreshing Terraform state prior to plan...

template_file.foo.1: Refreshing state... (ID: 2b9e553f514d0789ecbee4ec00710b11645aebe744bd3de99b1ab6af9130c1e8)
template_file.foo.0: Refreshing state... (ID: aabd55fb4b7b23363c246265d1c4daedc3798149a7230f1c082c8994aa0db39c)
template_file.with_splat: Refreshing state... (ID: 7f2cbd10b2517e514a9a7a6052f08a17e16b829df003f60e612b56f533e982dd)
template_file.no_splat: Refreshing state... (ID: 7f2cbd10b2517e514a9a7a6052f08a17e16b829df003f60e612b56f533e982dd)

<...>

+ template_file.foo.2
    rendered: "" => "<computed>"
    template: "" => "I AM 2"


Plan: 1 to add, 0 to change, 0 to destroy.

Of course, this reintroduces the bug from #2744, which you can see with this same example if we destroy and start over with an initial plan:

$ terraform plan
Refreshing Terraform state prior to plan...

<...>

+ template_file.foo.0
    rendered: "" => "<computed>"
    template: "" => "I AM 0"

+ template_file.foo.1
    rendered: "" => "<computed>"
    template: "" => "I AM 1"

+ template_file.no_splat
    rendered: "" => "<computed>"
    template: "" => "the first foo says ${template_file.foo.0.rendered}"

+ template_file.with_splat
    rendered: "" => "<computed>"
    template: "" => "the first foo says 74D93920-ED26-11E3-AC10-0800200C9A66"


Plan: 4 to add, 0 to change, 0 to destroy.

In this particular case the apply still worked as expected, but I understand this bug sometimes causes diffs to fail when they shouldn't, which is not good either.

Not sure where to go from here. In theory, reevaluating dependent values during the apply phase after each resource is created could solve for this, but Terraform seems intently designed to only ever evaluate these values during the plan phase. This is how plans can be generated and executed in a decoupled manner, giving the feel that you're applying a patch to your infrastructure just like you would to some C project. I get that. On the other hand, we're not talking about a Makefile needlessly recompiling a couple of libs, but active running components that are often critical to our production stacks. I feel like more precision is needed with these types of evaluations, but it's not immediately clear to me how to elegantly tackle this myself. Would love to hear others' opinions on the matter.

Oh, what a difference 3 lines of code makes!

[phinze]

Hey @justinclayton thanks for the great write up on this issue! I'm consolidating the open issues for this down to #3449 and I'll follow up there with a proposed plan of action. This example will be very useful going forward as absolutely clear documentation of the problem and the negative implications of a straight revert of #2744. 👍

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.