ignore_changes: diffs didn't match during apply

[glasser]

I got this with terraform apply. It happens moderately often on this ASG.

* aws_autoscaling_group.app: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.6.15
    Resource ID: aws_autoscaling_group.app
    Mismatch reason: extra attributes: termination_policies.0, wait_for_capacity_timeout, tag.#, delete_delay, initial_lifecycle_hook.#, vpc_zone_identifier.790767277, vpc_zone_identifier.2107437180, health_check_type, tag.1316750306.key, tag.1316750306.value, vpc_zone_identifier.1004301864, vpc_zone_identifier.2894510629, default_cooldown, health_check_grace_period, min_size, max_size, vpc_zone_identifier.#, termination_policies.#, metrics_granularity
    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"initial_lifecycle_hook.3307058257.name":*terraform.ResourceAttrDiff{Old:"glasser-app-7kjpxkokzzgpjdlld54fgxcyqa_launching", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "name":*terraform.ResourceAttrDiff{Old:"glasser-app-7kjpxkokzzgpjdlld54fgxcyqa", New:"${aws_launch_configuration.app.id}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, "tag.1316750306.propagate_at_launch":*terraform.ResourceAttrDiff{Old:"true", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "launch_configuration":*terraform.ResourceAttrDiff{Old:"glasser-app-7kjpxkokzzgpjdlld54fgxcyqa", New:"${aws_launch_configuration.app.id}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.~3327677310.name":*terraform.ResourceAttrDiff{Old:"", New:"${aws_launch_configuration.app.id}_launching", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.3307058257.notification_metadata":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.3307058257.notification_target_arn":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "force_delete":*terraform.ResourceAttrDiff{Old:"false", New:"0", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.3307058257.lifecycle_transition":*terraform.ResourceAttrDiff{Old:"autoscaling:EC2_INSTANCE_LAUNCHING", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.~3327677310.lifecycle_transition":*terraform.ResourceAttrDiff{Old:"", New:"autoscaling:EC2_INSTANCE_LAUNCHING", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.3307058257.role_arn":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.~3327677310.default_result":*terraform.ResourceAttrDiff{Old:"", New:"ABANDON", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.3307058257.heartbeat_timeout":*terraform.ResourceAttrDiff{Old:"600", New:"0", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.~3327677310.heartbeat_timeout":*terraform.ResourceAttrDiff{Old:"", New:"600", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"vpc_zone_identifier.790767277":*terraform.ResourceAttrDiff{Old:"", New:"subnet-64a52f3c", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.1914666129.default_result":*terraform.ResourceAttrDiff{Old:"", New:"ABANDON", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "termination_policies.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "vpc_zone_identifier.2107437180":*terraform.ResourceAttrDiff{Old:"", New:"subnet-d3f371f9", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.1914666129.name":*terraform.ResourceAttrDiff{Old:"", New:"glasser-app-4746vjkmuje6rcluhzc2rpulie_launching", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "health_check_type":*terraform.ResourceAttrDiff{Old:"", New:"EC2", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "name":*terraform.ResourceAttrDiff{Old:"", New:"glasser-app-4746vjkmuje6rcluhzc2rpulie", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, "initial_lifecycle_hook.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "launch_configuration":*terraform.ResourceAttrDiff{Old:"", New:"glasser-app-4746vjkmuje6rcluhzc2rpulie", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "delete_delay":*terraform.ResourceAttrDiff{Old:"", New:"60", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "tag.1316750306.propagate_at_launch":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "min_size":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "max_size":*terraform.ResourceAttrDiff{Old:"", New:"5", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "vpc_zone_identifier.#":*terraform.ResourceAttrDiff{Old:"", New:"4", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "termination_policies.0":*terraform.ResourceAttrDiff{Old:"", New:"OldestInstance", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.1914666129.heartbeat_timeout":*terraform.ResourceAttrDiff{Old:"", New:"600", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "initial_lifecycle_hook.1914666129.lifecycle_transition":*terraform.ResourceAttrDiff{Old:"", New:"autoscaling:EC2_INSTANCE_LAUNCHING", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "vpc_zone_identifier.1004301864":*terraform.ResourceAttrDiff{Old:"", New:"subnet-a3074a9e", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "tag.1316750306.key":*terraform.ResourceAttrDiff{Old:"", New:"Name", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "wait_for_capacity_timeout":*terraform.ResourceAttrDiff{Old:"", New:"100h", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "tag.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "force_delete":*terraform.ResourceAttrDiff{Old:"", New:"0", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "vpc_zone_identifier.2894510629":*terraform.ResourceAttrDiff{Old:"", New:"subnet-8b077afd", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "default_cooldown":*terraform.ResourceAttrDiff{Old:"", New:"15", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "health_check_grace_period":*terraform.ResourceAttrDiff{Old:"", New:"600", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "tag.1316750306.value":*terraform.ResourceAttrDiff{Old:"", New:"glasser-app", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "metrics_granularity":*terraform.ResourceAttrDiff{Old:"", New:"1Minute", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}

We are running TF from a relatively recent master with a couple (presumably irrelevant) local changes: meteor@4df071f

I have not been able to minimize this. This seems to only affect this ASG even though we have others; the most obvious difference is that this ASG has aws_autoscaling_policy and aws_cloudwatch_metric_alarm on it and the others don't.

[glasser]

OK, I've made my source files a lot smaller and this is good enough to do it (I can probably get it even smaller but it's the end of my day right now). I can create this successfully, but then if I change user_data and run apply again, I get the diff mismatch error. Then I run it again and it works.

(I haven't tried it without our wrapper script that does remote state, will try to do that later.)

[meh, see next comment for simpler]

When I delete the ignore_changes, it's OK.

[glasser]

And this is with master, built at c8928e4.

[glasser]

OK, here's entirely stripped down, built at c8928e4. This uses a capacity-zero ASG so it starts up fast!

A single tf file (with an AWS profile called dev):

# AWS credentials
provider "aws" {
  profile = "dev"
  region = "us-east-1"
}
resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "glasser-tf-test"
  }
}
resource "aws_subnet" "us-east-1a" {
  vpc_id = "${aws_vpc.main.id}"
  cidr_block = "10.0.0.0/24"
  availability_zone = "us-east-1a"
  tags = { Name = "glasser-tf-test-1a" }
}
resource "aws_launch_configuration" "app" {
  name_prefix = "glasser-tf-test-"
  lifecycle {
    create_before_destroy = true
  }
  image_id = "ami-7a190010"
  instance_type = "t2.micro"
  user_data = "foo"
}

resource "aws_autoscaling_group" "app" {
  name = "${aws_launch_configuration.app.id}"
  lifecycle {
    create_before_destroy = true
    ignore_changes = ["desired_capacity"]
  }
  max_size = "0"
  min_size = "0"
  desired_capacity = "0"
  launch_configuration = "${aws_launch_configuration.app.id}"
  vpc_zone_identifier = ["${aws_subnet.us-east-1a.id}"]
}

Run it, change user_data, run again:

glasser@glasser-lyrid 1 ~/Scratch/tf-bug master *#% $ terraform apply
aws_launch_configuration.app: Creating...
  associate_public_ip_address: "" => "0"
  ebs_block_device.#:          "" => "<computed>"
  ebs_optimized:               "" => "<computed>"
  enable_monitoring:           "" => "1"
  image_id:                    "" => "ami-7a190010"
  instance_type:               "" => "t2.micro"
  key_name:                    "" => "<computed>"
  name:                        "" => "<computed>"
  name_prefix:                 "" => "glasser-tf-test-"
  root_block_device.#:         "" => "<computed>"
  user_data:                   "" => "0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33"
aws_vpc.main: Creating...
  cidr_block:                "" => "10.0.0.0/16"
  default_network_acl_id:    "" => "<computed>"
  default_security_group_id: "" => "<computed>"
  dhcp_options_id:           "" => "<computed>"
  enable_classiclink:        "" => "<computed>"
  enable_dns_hostnames:      "" => "<computed>"
  enable_dns_support:        "" => "<computed>"
  main_route_table_id:       "" => "<computed>"
  tags.#:                    "" => "1"
  tags.Name:                 "" => "glasser-tf-test"
aws_launch_configuration.app: Creation complete
aws_vpc.main: Creation complete
aws_subnet.us-east-1a: Creating...
  availability_zone:       "" => "us-east-1a"
  cidr_block:              "" => "10.0.0.0/24"
  map_public_ip_on_launch: "" => "0"
  tags.#:                  "" => "1"
  tags.Name:               "" => "glasser-tf-test-1a"
  vpc_id:                  "" => "vpc-7de01d1a"
aws_subnet.us-east-1a: Creation complete
aws_autoscaling_group.app: Creating...
  availability_zones.#:         "" => "<computed>"
  default_cooldown:             "" => "<computed>"
  desired_capacity:             "" => "0"
  force_delete:                 "" => "0"
  health_check_grace_period:    "" => "<computed>"
  health_check_type:            "" => "<computed>"
  launch_configuration:         "" => "glasser-tf-test-bpjhs7rkgrbotaucobrnh4kuje"
  max_size:                     "" => "0"
  metrics_granularity:          "" => "1Minute"
  min_size:                     "" => "0"
  name:                         "" => "glasser-tf-test-bpjhs7rkgrbotaucobrnh4kuje"
  vpc_zone_identifier.#:        "" => "1"
  vpc_zone_identifier.17961702: "" => "subnet-922d43ca"
  wait_for_capacity_timeout:    "" => "10m"
aws_autoscaling_group.app: Creation complete

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate
glasser@glasser-lyrid 0 ~/Scratch/tf-bug master *#% $ perl -pi -e 's/foo/bar/' tf-bug.tf  # change user_data
glasser@glasser-lyrid 0 ~/Scratch/tf-bug master *#% $ terraform apply
aws_launch_configuration.app: Refreshing state... (ID: glasser-tf-test-bpjhs7rkgrbotaucobrnh4kuje)
aws_vpc.main: Refreshing state... (ID: vpc-7de01d1a)
aws_subnet.us-east-1a: Refreshing state... (ID: subnet-922d43ca)
aws_autoscaling_group.app: Refreshing state... (ID: glasser-tf-test-bpjhs7rkgrbotaucobrnh4kuje)
aws_launch_configuration.app: Creating...
  associate_public_ip_address: "" => "0"
  ebs_block_device.#:          "" => "<computed>"
  ebs_optimized:               "" => "<computed>"
  enable_monitoring:           "" => "1"
  image_id:                    "" => "ami-7a190010"
  instance_type:               "" => "t2.micro"
  key_name:                    "" => "<computed>"
  name:                        "" => "<computed>"
  name_prefix:                 "" => "glasser-tf-test-"
  root_block_device.#:         "" => "<computed>"
  user_data:                   "" => "62cdb7020ff920e5aa642c3d4066950dd1f01f4d"
aws_launch_configuration.app: Creation complete
Error applying plan:

1 error(s) occurred:

* aws_autoscaling_group.app: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.6.15
    Resource ID: aws_autoscaling_group.app
    Mismatch reason: extra attributes: vpc_zone_identifier.#, metrics_granularity, vpc_zone_identifier.17961702, min_size, wait_for_capacity_timeout, max_size
    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"name":*terraform.ResourceAttrDiff{Old:"glasser-tf-test-bpjhs7rkgrbotaucobrnh4kuje", New:"${aws_launch_configuration.app.id}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, "force_delete":*terraform.ResourceAttrDiff{Old:"false", New:"0", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "launch_configuration":*terraform.ResourceAttrDiff{Old:"glasser-tf-test-bpjhs7rkgrbotaucobrnh4kuje", New:"${aws_launch_configuration.app.id}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"vpc_zone_identifier.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "metrics_granularity":*terraform.ResourceAttrDiff{Old:"", New:"1Minute", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "min_size":*terraform.ResourceAttrDiff{Old:"", New:"0", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "wait_for_capacity_timeout":*terraform.ResourceAttrDiff{Old:"", New:"10m", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "force_delete":*terraform.ResourceAttrDiff{Old:"", New:"0", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "vpc_zone_identifier.17961702":*terraform.ResourceAttrDiff{Old:"", New:"subnet-922d43ca", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "name":*terraform.ResourceAttrDiff{Old:"", New:"glasser-tf-test-yz4ykkhjjzc7tdor5jpeedd7wq", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, "launch_configuration":*terraform.ResourceAttrDiff{Old:"", New:"glasser-tf-test-yz4ykkhjjzc7tdor5jpeedd7wq", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "max_size":*terraform.ResourceAttrDiff{Old:"", New:"0", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.


Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[phinze]

Thanks for providing the solid repro case @glasser! Digging into this now.

[phinze]

Just confirmed that this is fixed in v0.7.0. Thanks again for the clear report!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.