AWS route table and nat instance route modification

[igoratencompass]

Hi all,

I have a private route table resource in ec2 being created as:

resource "aws_route_table" "private-subnet" {
    vpc_id = "${aws_vpc.environment.id}"{
        Name        = "${var.vpc.tag}-private-subnet-route-table"
        Environment = "${var.vpc.tag}"
    }
}

resource "aws_route_table_association" "private-subnet" {
    count          = "${var.vpc.az_count}"
    subnet_id      = "${element(aws_subnet.private-subnets.*.id, count.index)}"
    route_table_id = "${aws_route_table.private-subnet.id}"
}

Then later in the same play, a nat instance gets launched that modifies the route table adding it's self as default gateway via user-data script. Now when ever I run the plan again after adding new resource lets say, TF modifies the route table removing the nat instance record.

Since TF is aware of the changes made by user-data of another resource, reverting the changes is a wrong thing to do.

I have asked about this in TF google group but never got any response back explaining why is this happening, please see https://groups.google.com/forum/#!topic/terraform-tool/MxEXo9hhqHk

Thanks,
Igor

[majormoses]

I believe I am doing the same thing you are (please correct me if I am wrong) in the userdata you are passing a bash script to be executed which changes the route table for the private subnet to point to its eni which it is in a public subnet with an internet gateway.

I could be wrong here but TF only knows about changes to the userdata itself not the result of the userdata run. I would not expect that TF to be aware of the route table changes and therefore will modify them to what you declared.

It would be nice if there was an option to allow specifying that a route table should be created once and not touched again (obviously unless you taint it) as not all route tables are static and the dynamic route is outside TF scope to know. I believe this is intended functionality and not a bug, that being said I think there is a clear path to allowing the behavior you are expecting and I am sure that we are not the only people that would benefit from said enhancement. In the short term if you need a work around you can do what I done in the past: applying, commenting out the route tables, and removing them from the statefile but that is obviously not ideal.

@catsby would love to hear your thoughts on this

[igoratencompass]

@majormoses , I guess the question what should and what should not be consider as a change is a debatable question. The bottom line is TF is aware of the change made to one resource by another one and should keep the change as valid. The same way that the changes made by aws_security_group_rule to aws_security_group resource should be honoured, otherwise we are heading towards a chaos, don't you think so :-)?

Regarding the workaround, yes it ia a viable one, although one that should not be needed in a product tending to be a production ready.

[igoratencompass]

Just as an example, the initial plan run:

"+" aws_route_table.private-subnet
tags.#: "" => "2"
tags.Environment: "" => "ENV"
tags.Name: "" => "ENV-private-subnet-route-table"
vpc_id: "" => "${aws_vpc.environment.id}"

The run:

aws_route_table.private-subnet: Creating...
tags.#: "" => "2"
tags.Environment: "" => "ENV"
tags.Name: "" => "ENV-private-subnet-route-table"
vpc_id: "" => "vpc-48a4f42d"

When finished I make a small change to a security group of some random resource and re-run the plan:

~ aws_route_table.private-subnet
route.#: "1" => "0"
route.2890251652.cidr_block: "0.0.0.0/0" => ""
route.2890251652.gateway_id: "" => ""
route.2890251652.instance_id: "i-bd274d36" => ""
route.2890251652.network_interface_id: "eni-a5eb7bed" => ""
route.2890251652.vpc_peering_connection_id: "" => ""

Obviously it is very much aware of the change and even gives me the details of the NAT instance, its id and the network interface.

[majormoses]

@igoratencompass I just tested with the latest version of TF v0.6.8 and I dont see this behavior anymore, maybe our prayers were silently answered?

[pikeas]

You may be running into #4097.

[igoratencompass]

@pikeas, thanks that looks exactly like what is happening in my case. So it is not what we were thinking is happening, it's a bug removing the network_interface_id no matter what.

[SpencerBrown]

No I don't think this has anything to do with issue #4097. What's happening is that Terraform is removing the entire route, not just the network interface ID. This is basic Terraform design, it believes it owns the resources, and will happily delete any additions you have made outside of Terraform.

I can think of a couple of solutions to this use case:

1. Rather than adding the route via a bash script in the userdata, add it via Terraform, specifying the instance ID of the NAT instance (which was also launched by Terraform).
2. use the ignore_changes feature to stop Terraform from changing the route table entries.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.