ebs_block_device conflicting with AMI block device mapping causing resource re-creation

[chrismyang]

Not sure if this is the right forum for these questions, so happy to re-post elsewhere!

---

We're experiencing an issue where we're launching an EC2 instance from an AMI that already has EBS volume snapshots in it, and the EC2 instance template has those EBS volumes specified.

Terraform wants to destroy and recreate the instance everytime we run apply with something like this message:

-/+ aws_instance.executor_template
    ami:                                               "ami-c2c421a2" => "ami-c2c421a2"
    availability_zone:                                 "us-west-2c" => "<computed>"
    ebs_block_device.#:                                "2" => "2"
    ebs_block_device.2173536061.delete_on_termination: "1" => "0"
    ebs_block_device.2173536061.device_name:           "xvds" => ""
    ebs_block_device.2804386727.delete_on_termination: "" => "1" (forces new resource)
    ebs_block_device.2804386727.device_name:           "" => "xvds" (forces new resource)
    ebs_block_device.2804386727.encrypted:             "" => "<computed>" (forces new resource)
    ebs_block_device.2804386727.iops:                  "" => "<computed>" (forces new resource)
    ebs_block_device.2804386727.snapshot_id:           "" => "<computed>" (forces new resource)
    ebs_block_device.2804386727.volume_size:           "" => "8" (forces new resource)
    ebs_block_device.2804386727.volume_type:           "" => "gp2" (forces new resource)
    ebs_block_device.3170548184.delete_on_termination: "1" => "0"
    ebs_block_device.3170548184.device_name:           "sdf" => ""
    ebs_block_device.3854285918.delete_on_termination: "" => "1" (forces new resource)
    ebs_block_device.3854285918.device_name:           "" => "sdf" (forces new resource)
    ebs_block_device.3854285918.encrypted:             "" => "<computed>" (forces new resource)
    ebs_block_device.3854285918.iops:                  "" => "<computed>" (forces new resource)
    ebs_block_device.3854285918.snapshot_id:           "" => "<computed>" (forces new resource)
    ebs_block_device.3854285918.volume_size:           "" => "340" (forces new resource)
    ebs_block_device.3854285918.volume_type:           "" => "gp2" (forces new resource)
    ebs_optimized:                                     "false" => "0"
    ephemeral_block_device.#:                          "0" => "<computed>"
    iam_instance_profile:                              "override-base-ami-executor" => "override-base-ami-executor"
    instance_type:                                     "t2.medium" => "t2.medium"
    key_name:                                          "override-base-ami-access-key" => "override-base-ami-access-key"
    placement_group:                                   "" => "<computed>"
    private_dns:                                       "ip-10-0-245-167.us-west-2.compute.internal" => "<computed>"
    private_ip:                                        "10.0.245.167" => "<computed>"
    public_dns:                                        "ec2-52-10-126-13.us-west-2.compute.amazonaws.com" => "<computed>"
    public_ip:                                         "52.10.126.13" => "<computed>"
    root_block_device.#:                               "1" => "1"
    root_block_device.0.delete_on_termination:         "true" => "1"
    root_block_device.0.iops:                          "90" => "<computed>"
    root_block_device.0.volume_size:                   "30" => "30"
    root_block_device.0.volume_type:                   "gp2" => "gp2"
    security_groups.#:                                 "0" => "<computed>"
    source_dest_check:                                 "true" => "1"
    subnet_id:                                         "subnet-aae2acf3" => "subnet-aae2acf3"
    tags.#:                                            "1" => "1"
    tags.Name:                                         "override-base-ami-executor-template" => "override-base-ami-executor-template"
    tenancy:                                           "default" => "<computed>"
    user_data:                                         "d947ed466806d5fdf040b1d567cbf31cedf79b3b" => "d947ed466806d5fdf040b1d567cbf31cedf79b3b"
    vpc_security_group_ids.#:                          "1" => "1"
    vpc_security_group_ids.1761259786:                 "sg-6f3dd508" => "sg-6f3dd508"

Conceptually, this seems like it ought not need to recreate the instance, because the EBS block devices really do have the matching properties. But it seems like TF can't track those EBS resources created through AMI launch process...

Here is an excerpt of the resource template:

resource "aws_instance" "executor_template" {

    ami = "${coalesce(var.override_executor_base_ami, var.default_executor_base_ami)}"

    root_block_device {
      volume_size = "${var.root_volume_size.executor_template}"
      volume_type = "gp2"
    }

    ebs_block_device {
      device_name = "sdf"
      volume_size = "${var.data_volume_size.executor_template}"
      delete_on_termination = true
      volume_type = "gp2"
      encrypted = "${var.should_encrypt_executor_ebs_volumes}"
    }

    ebs_block_device {
      device_name = "xvds"
      volume_size = "8"
      delete_on_termination = true
      volume_type = "gp2"
      encrypted = "${var.should_encrypt_executor_ebs_volumes}"
  }

  // Other fields
}  

Two other interesting things of note:

• If I remove the ebs_block_device blocks completely, then the initial launch and susquent apply converges happily -- no changes detected. Removing those block devices in general isn't ideal, and we couldn't figure out a way to get TF template interpolation to do what I needed. We'd like to avoid integration a pre-Terraform templating engine if possible, but if that's the only solution then...that would be good to know.
• Even if I specify ignore_changes through the lifecycle block, i.e.,

  lifecycle {
    ignore_changes = [
      "ebs_block_device"
    ]
  }

TF still detects the instance as needing to be destroyed and re-created, even though it no longer lists the ebs_block_device in the diff. Baffling!

-/+ aws_instance.executor_template
    ami:                                       "ami-a207e3c2" => "ami-a207e3c2"
    associate_public_ip_address:               "true" => "1"
    availability_zone:                         "us-west-2b" => "<computed>"
    ebs_optimized:                             "false" => "0"
    ephemeral_block_device.#:                  "0" => "<computed>"
    iam_instance_profile:                      "cmy-override-base-ami-1-executor" => "cmy-override-base-ami-1-executor"
    instance_type:                             "t2.medium" => "t2.medium"
    key_name:                                  "cmy-override-base-ami-1-access-key" => "cmy-override-base-ami-1-access-key"
    placement_group:                           "" => "<computed>"
    private_dns:                               "ip-10-0-139-151.us-west-2.compute.internal" => "<computed>"
    private_ip:                                "10.0.139.151" => "<computed>"
    public_dns:                                "ec2-52-32-57-149.us-west-2.compute.amazonaws.com" => "<computed>"
    public_ip:                                 "52.32.57.149" => "<computed>"
    root_block_device.#:                       "1" => "1"
    root_block_device.0.delete_on_termination: "true" => "1"
    root_block_device.0.iops:                  "90" => "<computed>"
    root_block_device.0.volume_size:           "30" => "30"
    root_block_device.0.volume_type:           "gp2" => "gp2"
    security_groups.#:                         "0" => "<computed>"
    source_dest_check:                         "true" => "1"
    subnet_id:                                 "subnet-c6120aa3" => "subnet-c6120aa3"
    tags.#:                                    "1" => "1"
    tags.Name:                                 "cmy-override-base-ami-1-executor-template" => "cmy-override-base-ami-1-executor-template"
    tenancy:                                   "default" => "<computed>"
    user_data:                                 "d0c4a1d093e69b24185170144390fbaec41a0889" => "d0c4a1d093e69b24185170144390fbaec41a0889"
    vpc_security_group_ids.#:                  "1" => "1"
    vpc_security_group_ids.1813430666:         "sg-fcbe4c9b" => "sg-fcbe4c9b"
Plan: 1 to add, 0 to change, 1 to destroy.

---

In case it helps, here is the whole reason we're trying to do this: the instance specified by the template is expensive to create because we end up installing a bunch of stuff that takes a long time, so we wanted to take an AMI snapshot of an already created one, and use that as a sort of "cached" instance that we can use for development/CI purposes. But obviously we won't use that cached AMI for our "real" deploys.

We're open to any solution to our use case, our though around this "override" base AMI was just the most promising path we saw initially. But definitely open to different ideas! Thanks so much in advance for your help! By and large, TF has been great!

---

I've found some related issues, but they seemed not quite the same or the resolution didn't quite work for what we're trying:

• #4345 ebs_block_device in aws_launch_configuration forces perpetual resource recreation seems to imply that the AMI and the template parameters are merged
• #3739 lifecycle with aws_instance causes diffs didn't match during apply seem to also discuss other issues with lifecycle but with different ignore_changes parameters (and an actual error, rather than just a divergence)
• #4786 Changes found in EBS Block device where there aren't really changes Similar issue, but the thread seemed to imply something about the EBS volume getting confused by the the root volume
• #3725 AWS: plan always out of date when AMI includes an EBS device Actually this looks like a duplicate!

[chrismyang]

Okay looks like there's a lot of prior related/duplicate questions. I'm wondering then if we could just get some guidance on the best workaround for now? I'm not sure if there are plans to resolve this issue (or if it's even possible) but we don't mind doing something a little hacky for now.

[chrismyang]

Ah the idea proposed in #3725 (comment) works for me -- if you specify snapshot_id then it happily converges. This works for what we need. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.