Routing table entry keeps being rebuilt

[josh-padnick]

Terraform Version

Terraform v0.6.16

Affected Resource(s)

• aws_route

Terraform Configuration Files

vpc-peering module:

# Create a VPC Peering Connection, and add routes in all subnets in both VPCs

resource "aws_vpc_peering_connection" "vpc_peering_connection" {
  peer_owner_id = "${var.aws_account_id}"
  vpc_id = "${var.origin_vpc_id}"
  peer_vpc_id = "${var.destination_vpc_id}"
  auto_accept = true
  tags { Name = "${var.origin_vpc_name}-to-${var.destination_vpc_name}" }
}

resource "aws_route" "origin_to_destination" {
  count = "${var.num_origin_vpc_route_tables}"
  route_table_id = "${element(split(",", var.origin_vpc_route_table_ids), count.index)}"
  destination_cidr_block = "${var.destination_vpc_cidr_block}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.vpc_peering_connection.id}"
}

resource "aws_route" "destination_to_origin" {
  count = "${var.num_destination_vpc_route_tables}"
  route_table_id = "${element(split(",", var.destination_vpc_route_table_ids), count.index)}"
  destination_cidr_block = "${var.origin_vpc_cidr_block}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.vpc_peering_connection.id}"
}

top-level terraform template:

module "mgmt_to_temp" {
  source = "git::[email protected]:gruntwork-io/module-vpc.git//modules/vpc-peering?ref=v0.0.3"

  aws_account_id = "..."

  origin_vpc_id = "${terraform_remote_state.mgmt_vpc.output.vpc_id}"
  origin_vpc_name = "${terraform_remote_state.mgmt_vpc.output.vpc_name}"
  origin_vpc_cidr_block = "${terraform_remote_state.mgmt_vpc.output.vpc_cidr_block}"
  origin_vpc_route_table_ids = "${terraform_remote_state.mgmt_vpc.output.private_subnet_route_table_ids},${terraform_remote_state.mgmt_vpc.output.public_subnet_route_table_id}"
  # We should be able to compute these numbers automatically, but can't due to https://github.com/hashicorp/terraform/issues/3888.
  num_origin_vpc_route_tables = "${length(split(",", var.aws_availability_zones)) + 1}"

  destination_vpc_id = "${module.temp_vpc.vpc_id}"
  destination_vpc_name = "${module.temp_vpc.vpc_name}"
  destination_vpc_cidr_block = "${module.temp_vpc.vpc_cidr_block}"
  destination_vpc_route_table_ids = "${module.temp_vpc.public_subnet_route_table_id},${module.temp_vpc.private_app_subnet_route_table_ids},${module.temp_vpc.private_persistence_route_table_ids}"
  num_destination_vpc_route_tables = "${(length(split(",", var.aws_availability_zones)) * 2) + 1}"
}

Expected Behavior

Terraform correctly creates these resources on the first terraform apply. Repeated use of terraform plan and terraform apply should show no change in resources.

Actual Behavior

Terraform correctly creates these resources on the first terraform apply. Without modifying the templates, when I run terraform plan I get:

~ module.temp_vpc.aws_route_table.public
    route.#:                                    "2" => "1"
    route.1673710610.cidr_block:                "10.10.0.0/18" => ""
    route.1673710610.gateway_id:                "" => ""
    route.1673710610.instance_id:               "" => ""
    route.1673710610.nat_gateway_id:            "" => ""
    route.1673710610.network_interface_id:      "" => ""
    route.1673710610.vpc_peering_connection_id: "pcx-c99524a0" => ""
    route.4214658771.cidr_block:                "0.0.0.0/0" => "0.0.0.0/0"
    route.4214658771.gateway_id:                "igw-e046a184" => "igw-e046a184"
    route.4214658771.instance_id:               "" => ""
    route.4214658771.nat_gateway_id:            "" => ""
    route.4214658771.network_interface_id:      "" => ""
    route.4214658771.vpc_peering_connection_id: "" => ""

I terraform apply this, it completes successfully. Then I run terraform plan again and terraform wants to re-create the route it just destroyed:

+ module.mgmt_to_temp.aws_route.destination_to_origin.0
    destination_cidr_block:     "" => "10.10.0.0/18"
    destination_prefix_list_id: "" => "<computed>"
    gateway_id:                 "" => "<computed>"
    instance_id:                "" => "<computed>"
    instance_owner_id:          "" => "<computed>"
    nat_gateway_id:             "" => "<computed>"
    network_interface_id:       "" => "<computed>"
    origin:                     "" => "<computed>"
    route_table_id:             "" => "rtb-631c4207"
    state:                      "" => "<computed>"
    vpc_peering_connection_id:  "" => "pcx-c99524a0"

This behavior continues indefinitely.

Important Factoids

We're setting up Route Tables in the context of VPC Peering to connect multiple different VPCs. Each VPC's terraform state is stored in a separate .tfstate file in S3.

References

• Routing table entry and security groups keeps being rebuilt #5423 (probably the same issue)
• Diffs when there shouldn't be... #6832 (similar, but not the same problem)

[jrnt30]

Are you defining any routes in the default aws_route_table resource? Based upon the log it seems like you may be.

If you are, take a look at the "note" on the aws_route_table resource https://www.terraform.io/docs/providers/aws/r/route_table.html

[stack72]

Hi @josh-padnick

What @jrnt30 is indeed correct - aws_route and aws_route_table with routes in it compete against each other. This is why we get the remove then readd behaviour

this is a known limitation

Paul

[josh-padnick]

@stack72 @jrnt30 You guys are exactly correct! In our multi-layered templates, we didn't realize we'd done this. Thanks so much for the prompt responses! Closing this now.

[FransUrbo]

Could this be reopened with a documentation tag added?

I was just struck with this as well and after reading the doc referenced above (#7303 (comment)) and this issue several times, I finaly managed to figured out how to solve this.

If I could suggest that something like this be added to the doc page to make it (more) obvious:

That is, creating a resource like this:

resource "aws_route_table" "test" {
  vpc_id                      = "${aws_vpc.test.id}"

  route {
    cidr_block                = "0.0.0.0/0"
    gateway_id                = "${aws_internet_gateway.test.id}"
  }
}

and then reference it with:

resource "aws_route" "peering_test" {
  route_table_id              = "${aws_route_table.test.id}"
  destination_cidr_block      = "${aws_vpc.test.cidr_block}"
  vpc_peering_connection_id   = "${aws_vpc_peering_connection.test.id}"
}

will make the two resources being recreated every time.

Instead, remove the route { } block from the aws_route_table resource and create a separate aws_route resource:

# An 'empty' route table so that the "aws_route"s below have something to 'bind' to.
resource "aws_route_table" "test" {
  vpc_id                      = "${aws_vpc.test.id}"
}

# The 'route { }' part of the original route table.
resource "aws_route" "test" {
  route_table_id              = "${aws_route_table.test.id}"
  destination_cidr_block      = "0.0.0.0/0"
  gateway_id                  = "${aws_internet_gateway.test.id}"
}

# The original route entry.
resource "aws_route" "peering_test" {
  route_table_id              = "${aws_route_table.test.id}"
  destination_cidr_block      = "${aws_vpc.test.cidr_block}"
  vpc_peering_connection_id   = "${aws_vpc_peering_connection.test.id}"
}

[stack72]

Hi @FransUrbo

This is already documented https://www.terraform.io/docs/providers/aws/r/route.html and https://www.terraform.io/docs/providers/aws/r/route_table.html

NOTE on Route Tables and Routes: Terraform currently provides both a standalone Route resource and a Route Table resource with routes defined in-line. At this time you cannot use a Route Table with in-line routes in conjunction with any Route resources. Doing so will cause a conflict of rule settings and will overwrite rules.

Hope this is sufficient?

P.

[FransUrbo]

A better idea would be to create a Caveats at the very bottom with the additional information.

That way that Caveats topic could reoccur on all pages to clearify things that needs clarifying.

[stack72]

@FransUrbo we use the Note syntax for that - it usually happens at the top of every page that requires it. Launch Config and AutoScaling groups are another area we do this on

[FransUrbo]

Just a suggestion. I'm sure there are cases where there's need for "more verbose notes" than the ones at the top. A Caveats would fit that perfectly I think.

[FransUrbo]

On Oct 25, 2016, at 1:25 PM, Paul Stack wrote:

This is already documented https://www.terraform.io/docs/providers/aws/r/route.html and https://www.terraform.io/docs/providers/aws/r/route_table.html

Yes, I know. What I'm saying is that I don't think it's obvious enough.

[dwcramer]

Why support inline routes at all if using them causes the resource to be recreated every time? I'm assuming it has to be this way for some reason. I'd rather have a warning or error from terraform telling me I'm doing something dumb than have it quietly let me do something that will cause problems later.

[catsby]

Why support inline routes at all if using them causes the resource to be recreated every time?

Using inline routes does not cause the resource to be created every time... using both inline routes and external routes (via aws_route) will.

Originally you could only define routes inline. The resource was designed much like most others: here is the resource, this is what it should look like, and only this. If you detect different, tell me.

Over time, especially with the introduction of modules, it became necessary to define a route in a module (or otherwise external one) and append routes to it as needed. Naturally, with the above defined behavior, this wouldn't work, we couldn't add rules later unless it was directly to the module. So, a compromise was made: one or the other. With this, you can export a route table id from the module, and consumers of the module can append rules to it, but can only remove rules said consumer knows about.

[jasonrogena]

I noticed some wonky behaviour when using aws_route, instead of a route block inside aws_route_table. Terraform (version 0.10.2) always failed on the first try, but passed on subsequent retries. Adding a depends_on in the route didn't fix the issue. This is the error I got:

InvalidParameterValue: There is no route defined for 0.0.0.0/0 in the route table. Use CreateRoute instead.

[josh-padnick]

@jasonrogena That is most likely an AWS eventual consistency issue. Terraform created the route successfully, then queries AWS to find its ID. But AWS returned nothing because the route had not yet propagated through all of AWS's API endpoints. The solution here is just to re-run terraform apply.

Note that, in my team's experience, you're more likely to experience these issues when using an AWS region that's physically far away from you, though you can still experience them when using the closest region.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.