resource/aws_route_table_association: error reading Route Table Association (rtbassoc-xxx): Empty result

[ialidzhikov]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

terraform version - 0.12.31
provider-aws version - 3.54.0

Affected Resource(s)

• aws_route_table_association

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

    provider "aws" {
      access_key = "${var.ACCESS_KEY_ID}"
      secret_key = "${var.SECRET_ACCESS_KEY}"
      region     = "eu-west-1"
    }

    resource "aws_vpc_dhcp_options" "vpc_dhcp_options" {
      domain_name         = "eu-west-1.compute.internal"
      domain_name_servers = ["AmazonProvidedDNS"]
    }

    resource "aws_vpc" "vpc" {
      cidr_block           = "10.250.0.0/16"
      enable_dns_support   = true
      enable_dns_hostnames = true
    }

    resource "aws_vpc_dhcp_options_association" "vpc_dhcp_options_association" {
      vpc_id          = "${aws_vpc.vpc.id}"
      dhcp_options_id = "${aws_vpc_dhcp_options.vpc_dhcp_options.id}"
    }

    resource "aws_default_security_group" "default" {
      vpc_id = "${aws_vpc.vpc.id}"
    }

    resource "aws_internet_gateway" "igw" {
      vpc_id = "${aws_vpc.vpc.id}"
    }

    resource "aws_route_table" "routetable_main" {
      vpc_id = "${aws_vpc.vpc.id}"
    }

    resource "aws_route" "public" {
      route_table_id         = "${aws_route_table.routetable_main.id}"
      destination_cidr_block = "0.0.0.0/0"
      gateway_id             = "${aws_internet_gateway.igw.id}"
    }

    resource "aws_security_group" "nodes" {
      name        = "foo-nodes"
      description = "Security group for nodes"
      vpc_id      = "${aws_vpc.vpc.id}"
    }

    resource "aws_security_group_rule" "nodes_self" {
      type              = "ingress"
      from_port         = 0
      to_port           = 0
      protocol          = "-1"
      self              = true
      security_group_id = "${aws_security_group.nodes.id}"
    }

    resource "aws_security_group_rule" "nodes_tcp_all" {
      type              = "ingress"
      from_port         = 30000
      to_port           = 32767
      protocol          = "tcp"
      cidr_blocks       = ["0.0.0.0/0"]
      security_group_id = "${aws_security_group.nodes.id}"
    }

    resource "aws_security_group_rule" "nodes_udp_all" {
      type              = "ingress"
      from_port         = 30000
      to_port           = 32767
      protocol          = "udp"
      cidr_blocks       = ["0.0.0.0/0"]
      security_group_id = "${aws_security_group.nodes.id}"
    }

    resource "aws_security_group_rule" "nodes_egress_all" {
      type              = "egress"
      from_port         = 0
      to_port           = 0
      protocol          = "-1"
      cidr_blocks       = ["0.0.0.0/0"]
      security_group_id = "${aws_security_group.nodes.id}"
    }


    resource "aws_subnet" "nodes_z0" {
      vpc_id            = "${aws_vpc.vpc.id}"
      cidr_block        = "10.250.0.0/19"
      availability_zone = "eu-west-1c"
    }

    output "subnet_nodes_z0" {
      value = "${aws_subnet.nodes_z0.id}"
    }

    resource "aws_subnet" "private_utility_z0" {
      vpc_id            = "${aws_vpc.vpc.id}"
      cidr_block        = "10.250.112.0/22"
      availability_zone = "eu-west-1c"
    }

    resource "aws_security_group_rule" "nodes_tcp_internal_z0" {
      type              = "ingress"
      from_port         = 30000
      to_port           = 32767
      protocol          = "tcp"
      cidr_blocks       = ["10.250.112.0/22"]
      security_group_id = "${aws_security_group.nodes.id}"
    }

    resource "aws_security_group_rule" "nodes_udp_internal_z0" {
      type              = "ingress"
      from_port         = 30000
      to_port           = 32767
      protocol          = "udp"
      cidr_blocks       = ["10.250.112.0/22"]
      security_group_id = "${aws_security_group.nodes.id}"
    }

    resource "aws_subnet" "public_utility_z0" {
      vpc_id            = "${aws_vpc.vpc.id}"
      cidr_block        = "10.250.96.0/22"
      availability_zone = "eu-west-1c"
    }

    output "subnet_public_utility_z0" {
      value = "${aws_subnet.public_utility_z0.id}"
    }

    resource "aws_security_group_rule" "nodes_tcp_public_z0" {
      type              = "ingress"
      from_port         = 30000
      to_port           = 32767
      protocol          = "tcp"
      cidr_blocks       = ["10.250.96.0/22"]
      security_group_id = "${aws_security_group.nodes.id}"
    }

    resource "aws_security_group_rule" "nodes_udp_public_z0" {
      type              = "ingress"
      from_port         = 30000
      to_port           = 32767
      protocol          = "udp"
      cidr_blocks       = ["10.250.96.0/22"]
      security_group_id = "${aws_security_group.nodes.id}"
    }

    resource "aws_eip" "eip_natgw_z0" {
      vpc = true
    }

    resource "aws_nat_gateway" "natgw_z0" {
      allocation_id = "${aws_eip.eip_natgw_z0.id}"
      subnet_id     = "${aws_subnet.public_utility_z0.id}"
    }

    resource "aws_route_table" "routetable_private_utility_z0" {
      vpc_id = "${aws_vpc.vpc.id}"
    }

    resource "aws_route" "private_utility_z0_nat" {
      route_table_id         = "${aws_route_table.routetable_private_utility_z0.id}"
      destination_cidr_block = "0.0.0.0/0"
      nat_gateway_id         = "${aws_nat_gateway.natgw_z0.id}"

      timeouts {
        create = "5m"
      }
    }

    resource "aws_route_table_association" "routetable_private_utility_z0_association_private_utility_z0" {
      subnet_id      = "${aws_subnet.private_utility_z0.id}"
      route_table_id = "${aws_route_table.routetable_private_utility_z0.id}"
    }

    resource "aws_route_table_association" "routetable_main_association_public_utility_z0" {
      subnet_id      = "${aws_subnet.public_utility_z0.id}"
      route_table_id = "${aws_route_table.routetable_main.id}"
    }

    resource "aws_route_table_association" "routetable_private_utility_z0_association_nodes_z0" {
      subnet_id      = "${aws_subnet.nodes_z0.id}"
      route_table_id = "${aws_route_table.routetable_private_utility_z0.id}"
    }


    //=====================================================================
    //= IAM instance profiles
    //=====================================================================

    resource "aws_iam_role" "bastions" {
      name = "foo-bastions"
      path = "/"

      assume_role_policy = <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ec2.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    EOF
    }

    resource "aws_iam_instance_profile" "bastions" {
      name = "foo-bastions"
      role = "${aws_iam_role.bastions.name}"
    }

    resource "aws_iam_role_policy" "bastions" {
      name = "foo-bastions"
      role = "${aws_iam_role.bastions.id}"

      policy = <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "ec2:DescribeRegions"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
    }
    EOF
    }

    resource "aws_iam_role" "nodes" {
      name = "foo-nodes"
      path = "/"

      assume_role_policy = <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ec2.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    EOF
    }

    resource "aws_iam_instance_profile" "nodes" {
      name = "foo-nodes"
      role = "${aws_iam_role.nodes.name}"
    }

    resource "aws_iam_role_policy" "nodes" {
      name = "foo-nodes"
      role = "${aws_iam_role.nodes.id}"

      policy = <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "ec2:DescribeInstances"
          ],
          "Resource": [
            "*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "ecr:GetAuthorizationToken",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:GetRepositoryPolicy",
            "ecr:DescribeRepositories",
            "ecr:ListImages",
            "ecr:BatchGetImage"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
    }
    EOF
    }

    //=====================================================================
    //= EC2 Key Pair
    //=====================================================================

    resource "aws_key_pair" "kubernetes" {
      key_name   = "foo-ssh-publickey"
      public_key = "ssh-rsa bar"
    }

    //=====================================================================
    //= Output variables
    //=====================================================================

    output "vpc_id" {
      value = "${aws_vpc.vpc.id}"
    }

    output "iamInstanceProfileNodes" {
      value = "${aws_iam_instance_profile.nodes.name}"
    }

    output "keyName" {
      value = "${aws_key_pair.kubernetes.key_name}"
    }

    output "security_group_nodes" {
      value = "${aws_security_group.nodes.id}"
    }

    output "nodes_role_arn" {
      value = "${aws_iam_role.nodes.arn}"
    }

Debug Output

Panic Output

Expected Behavior

Actual Behavior

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 3.54"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

aws_key_pair.kubernetes: Creating...
### Creation logs are omitted

* error reading Route Table Association (rtbassoc-xxx): Empty result
  on tf/main.tf line 374, in resource "aws_route_table_association" "routetable_private_utility_z1_association_nodes_z1":
    374: resource "aws_route_table_association" "routetable_private_utility_z1_association_nodes_z1" {

Steps to Reproduce

1. terraform apply the configuration from above

2. Make sure that for heavily used AWS account, it may fail with the above error:

* error reading Route Table Association (rtbassoc-xxx): Empty result
  on tf/main.tf line 374, in resource "aws_route_table_association" "routetable_private_utility_z1_association_nodes_z1":
    374: resource "aws_route_table_association" "routetable_private_utility_z1_association_nodes_z1" {

Maybe also cloud provider request limits and throttling can lead to this error?
Can this issue be related to the eventual consistency model of the AWS EC2 API (hence related to #16796)?

Important Factoids

References

• #0000

[mgusiew-guide]

I experience the same behavior using TF 0.13.7 and AWS provider 3.62.0.

Before upgrading to 3.62.0 I saw the problem with route tables which is now fixed - #19985

Now the route table problem is gone but still often get errors for route table associations as described above when running tests. Note that this issue is nondeterministic and may take few runs to reproduce. Below sample error:

Error:      	Received unexpected error:
        	            	FatalError{Underlying: error while running command: exit status 1; �[31m
        	            	�[1m�[31mError: �[0m�[0m�[1merror reading Route Table Association (rtbassoc-056487281f1396e9b): empty result�[0m
        	            	
        	            	�[0m  on .terraform/modules/basic-setup.vpc.vpc.vpc/main.tf line 1187, in resource "aws_route_table_association" "public":
        	            	1187: resource "aws_route_table_association" "public" �[4m{�[0m
        	            	�[0m
        	            	�[0m�[0m}

[cdancy]

We're hitting this as well, seemingly randomly, on version 3.63.0.

[anGie44]

Hi all 👋 the PR #21710 has been merged to hopefully address this nondeterministic issue. any findings from those who upgrade to the new provider that will be out later today (v3.65.0) would be greatly appreciated!

[mgusiew-guide]

Hi @anGie44 , unfortunately this issue is still present in v3.65.0. I analysed the code and it seems to me that FindRouteTableAssociationByID function is a problem. My suspicion is that it starts returning expected results in StatusRouteTableAssociationState check in WaitRouteTableAssociationUpdated but then for some reason it is not returning same results in subsequent resourceRouteTableAssociationRead. So either there is some bug in this function (in such case this needs to be fixed) or AWS is returning inconsistent results (in that case maybe WaitRouteTableAssociationUpdated should wait for more than one successful find invocation).

BTW it looks like there is IGW regression in v3.65.0 which affects Terraform destroy, I reported it as a separate issue: #21792

[mgusiew-guide]

FTR I tested this on 3.66.0 version and now I see routeTable issues like below (IIRC I saw route_table_association issues but much less often than routeTable):

[1m�[31mError: �[0m�[0m�[1merror reading Route Table (rtb-0fa601bb520958930): couldn't find resource�[0m

            	            	�[0m  on .terraform/modules/target-source-connection.external-vpc.vpc.vpc/main.tf line 203, in resource "aws_route_table" "public":
            	            	 203: resource "aws_route_table" "public" �[4m{�[0m
            	            	�[0m
            	            	�[0m�[0m} 

For now I decided to rollback to 3.62.0 on prod, seems to me that route table issues occur less often on that version.

[Amokrane2018]

I am experiencing same issue. I am using pulumi 4.31.0, itself using aws terraform provider 3.68.0
aws:ec2:RouteTableAssociation (myservice-subnet-1): error: 1 error occurred: * creating urn:pulumi:dev::myproject::awsx:x:ec2:Vpc$awsx:x:ec2:Subnet$aws:ec2/routeTableAssociation:RouteTableAssociation::myservice-subnet-1: 1 error occurred: * error reading Route Table Association (rtbassoc-xxx): empty result

[artem-nefedov]

Seeing the same issue on v3.71.0. It happens rarely, but we run a lot of tests, and encountered this multiple times already.

[cdancy]

@artem-nefedov same. Not sure what your setup looks like but we've found that wrapping the process in our own retry blocks seems to give better results. Unfortunate we have to do so but we also can't sit idly by waiting for this type of thing to be fixed which has been on-going seemingly forever now.

[jasonkinsella]

@cdancy This is also killing us. Could you share an example of the retry blocks so I can share with my devops? Are you also getting this - #22420 ?

[cdancy]

@jasonkinsella the retry blocks are not being done within terraform but outside using a python script. Basically we are retrying applying the terraform over and over again until it succeeds. It's not a great solution but it mostly works.

[jasonkinsella]

@cdancy Ha - yes that's exactly what my team implemented! Thought you might have something inside TF we'd missed. I'm hopeful #22420 related and all will be fixed in v3.73.0

[cdancy]

@jasonkinsella you and me both. These errors have taken so many different forms and variations and have been really really really frustrating for us the past 6 months or so.

[mgusiew-guide]

FTR I tried to briefly analyse that problem a little bit some time ago (version 3.66.0). It manifests either as empty result for route table lookup or route table association lookup. Both use same read API (route table find). What happens is that during create, route is created, then create waits until read returns the route (this succeeds) and then read is invoked once again to read resource but for some reason the query does not return it. So it seems to me that it is either AWS that is inconsistent with routes query or there is some issue in AWS provider that reads AWS read response inconsistently (e.g. pagination or sth else).

[artem-nefedov]

@mgusiew-guide From official AWS documentation

Eventual consistency

The Amazon EC2 API follows an eventual consistency model, due to the distributed nature of the system supporting the API. This means that the result of an API command you run that affects your Amazon EC2 resources might not be immediately visible to all subsequent commands you run. You should keep this in mind when you carry out an API command that immediately follows a previous API command.

Basically, this is not a bug from AWS perspective, and it's up to client to implement retries.

[mgusiew-guide]

@artem-nefedov AFAIR wait has retries and waits for the result to appear, the problem is that once wait succeeds there is another read and this read does not find the route. So it is a bit like DNS propagation, after it returns first correct result, it may happen then it will still return stale results for a while (this is due to distributed nature of the service). I would expect that once route read query returns the specific route, the subsequent read query will also return it. For some reason that is not the case. Note that I haven't analysed it deeply so my theory may be wrong. To verify this it would be good to setup provider in trace mode where aws commands are logged and try to reproduce. Unfortunately right now I don't have capacity to do that.

[artem-nefedov]

@mgusiew-guide

the problem is that once wait succeeds there is another read and this read does not find the route

Yes, that's exactly what "eventual consistency" means. Suppose you have a cluster of 3 nodes handling your request, and perform a write operation. That request goes to one of the nodes, which then asynchronously propagate changes to other nodes (which take some time). During this time, you can make read request, and it may go to original node (which you wrote to), and it will return data successfully. Then, you can make a second request, and it can potentially go to another node, which potentially doesn't have the data propagated yet, so you will get the failure.

This is exactly why the fix for similar problem with Security Groups says:

resource/aws_security_group: Ensure that the Security Group is found 3 times in a row before declaring that it has been created
https://github.com/hashicorp/terraform-provider-aws/pull/22420/files

[mgusiew-guide]

@artem-nefedov I agree with your train of thought. I think that adding ContinuousTargetOccurence: 3 (or higher value) to resource.StateChangeConf in WaitRouteTableAssociationCreated and WaitRouteTableReady in ec2/wait.go could solve this issue. Unfortunately I don't see that change in main branch. Hopefully somebody will pick it up and this gets resolved...

[mgusiew-guide]

I am sorry WaitRouteTableReady was not affected.

Just noticed that ContinuousTargetOccurence: 2 is already present in WaitRouteReady since 11/06/2021.
I am not sure if it is also needed in WaitRouteTableAssociationCreated, also not sure if 2 is high enough.

Hopefully someone will have a closer look.

[mritalian]

Apologies for the noise; in case it's helpful: this still happens on 3.72.0 and 3.73.0.

[mritalian]

This is a far from negligible occurrence pattern; in a dozen apply across 2 days I've seen it fail twice already (earlier it felt very rare). This impacts terraform-aws-modules/vpc/aws directly. And obviously completely breaks any CD pipelines that create and immediately manipulate route tables (i.e. everyone who creates a vpc?). It is so breaking in fact I have to ask myself if aws changed something in their backend to make this happen more often.

Bumping up literals like ContinuousTargetOccurence from 2 to 3 feels arbitrary. If it fixes it great, I guess, but I think we'd all rather not be reading the same discussion in a year when amazon scales out their backend further.

Retrying the entire script is like using a cannon as a fly swatter.

I'd be glad to test any potential fixes as I'm running into this on the daily.

[cdancy]

@mritalian you said what we're all thinking brother ... but they seem to only want to provide Band-Aids and not complete solutions

[jasonkinsella]

In the past 3 months we have gone from almost zero failures to about 5 per day across about 100 deployments. We had our provider pinned to 3.64 during this whole time. We've now got a dedicated customer channel reporting launch failures! I suspect there has been some AWS backend change that's amplified this weakness.

[mgusiew-guide]

Disclaimers:

1. As of today I am not AWS terraform provider contributor
2. Unfortunately these days I work on sth else and don't have time to dig deeper into this issue
3. One of my projects is affected by this so I am looking for some solution

Now some comments:

1. IMHO longterm it would be best if the eventual consistency issues were solved on AWS API level. There is wait API for a lot. of things (https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/wait/index.html) but not for routes. So those of us who have contact with AWS could ask for such an enhancement
2. If my suspicion that this issue is caused by inconsistent read holds, then I personally cant think of better short term solution than polling with ContinuousTargetOccurence. That is what I do when checking that DNS propagated properly, the only difference is that I check for 10 consecutive successful results before moving forward. So maybe increasing the counter could help. We will not know it for sure unless we have proof (see 3)
3. Maybe somebody could set up a trace (see https://www.terraform.io/cli/config/environment-variables) and try to reproduce and then share part of the logs or at least confirm hyphotesis (I understand some logs may contain sensitive data). I hope that once the hyphotesis is confirmed, somebody will step up and increase the ContinuousTargetOccurence parameter or make it configurable

FTR I agree that neither Terraform and Terraform AWS is perfect, there are gaps in coverage and that results in regressions. However, what I noticed is that when I report the problem and provide the details, somebody will look at it, e.g. #21792 (comment)

So maybe as a users we could a at least try to collect some more data in order to move this one forward.

[github-actions]

This functionality has been released in v4.0.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[kjenney]

v4.0.0 has breaking changes. Would it be possible to mitigate this in a v3.x version?

[mgusiew-guide]

FTR I ran my test suite on 4.0.0 (fortunately I am not affected by backwards incompatible changes) and I still receive route table errors like the one below (I use VPC module from terraform-aws-modules). Therefore I am not sure if this issue is completely fixed.

�[0m
�[0m�[0m�[0m
�[31m
�[1m�[31mError: �[0m�[0m�[1merror reading Route Table (rtb-0xxxxxxx): couldn't find resource�[0m

�[0m on .terraform/modules/vpc.vpc.vpc/main.tf line 203, in resource "aws_route_table" "public":
203: resource "aws_route_table" "public" �[4m{�[0m
�[0m
�[0m�[0m}

[bhadrim]

Hi, I am using hashicorp/aws v4.9.0 and I see the same error when I try to create aws_route_table_association

error reading Route Table Association (rtbassoc-xxxxxxxxxxxx): empty result

I see my route table , associated subnets and subnet associations created on AWS.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.