Add support for secondary_allocation_ids on nat gateways

[darkxeno]

Is your request related to a new offering from AWS?

Is this functionality available in the [AWS provider for Terraform]
(https://registry.terraform.io/providers/hashicorp/aws/latest/docs)? See CHANGELOG.md, too.

• No 🛑: please wait to file a request until the functionality is avaialble in the AWS provider
• Yes ✅: please list the AWS provider version which introduced this functionality

Yes: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/nat_gateway#public-nat-with-secondary-private-ip-addresses

Is your request related to a problem? Please describe.

I need to add secondary public ips to my NAT gateways in order to scale my egress bandwidth.

https://aws.amazon.com/blogs/networking-and-content-delivery/attach-multiple-ips-to-a-nat-gateway-to-scale-your-egress-traffic-pattern/

Describe the solution you'd like.

Add an input variable, of type list where an user can enter an arbitrary number of secondary allocation ids, the list length should match the number of NAT devices selected by the user, each list element references 1 to 7 EIPs, similarly to the current input field: external_nat_ip_ids

A maximum of 8 IPs can be associated with each NAT gateway, 1 primary and 7 secondary ones.

Describe alternatives you've considered.

There are no alternatives as my NAT gateways and VPC are managed by this terraform module. Maybe adding a lifecycle ignore on the secondary_allocation_ids field of the VPC module?

Additional context

[pdrastil]

I've tried to add this capability as I have similar need, however I'm running into following issue:

│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for aws_nat_gateway.this[2] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .association_id: was
│ known, but now unknown.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Could some of the maintainers take a look what might be wrong with this piece of code? Maybe there might be a better way to write this. When code is applied for the second time EIPs that have been created are also correctly associated with NAT Gateway.

Inputs:

variable "secondary_nat_ips" {
  description = "The number of secondary private IPv4 addresses assigned to NAT Gateways"
  type        = number
  default     = 0
}

main.tf

locals {
  nat_gateway_count         = var.single_nat_gateway ? 1 : var.one_nat_gateway_per_az ? length(var.azs) : local.max_subnet_length
  nat_gateway_ips           = var.reuse_nat_ips ? var.external_nat_ip_ids : try(aws_eip.nat[*].id, [])
  nat_gateway_secondary_ips = try(chunklist(aws_eip.nat_secondary[*].id, local.nat_gateway_count), [])
}

resource "aws_eip" "nat_secondary" {
  count = local.create_vpc && var.enable_nat_gateway && var.secondary_nat_ips > 0 ? local.nat_gateway_count * var.secondary_nat_ips : 0

  domain = "vpc"

  tags = merge(
    {
      "Name" = format(
        "${var.name}-%s-ext-%s",
        element(var.azs, var.single_nat_gateway ? 0 : count.index),
        floor(count.index / (var.single_nat_gateway ? 1 : length(var.azs))) + 1
      )
    },
    var.tags,
    var.nat_eip_tags,
  )

  depends_on = [aws_internet_gateway.this]
}

resource "aws_nat_gateway" "this" {
  count = local.create_vpc && var.enable_nat_gateway ? local.nat_gateway_count : 0

  allocation_id = element(
    local.nat_gateway_ips,
    var.single_nat_gateway ? 0 : count.index,
  )

  secondary_allocation_ids = [ 
    for id in local.nat_gateway_secondary_ips : element(id, count.index) 
  ]
  // omitted for brevity
}

[darkxeno]

Thank you @pdrastil for the help. I just tried the implementation you provided on the fork, but i am getting another error too, when trying to update a NAT gateway that had already 1 public ip set as primary.

module.networking.module.vpc.aws_nat_gateway.this[0]: Still creating... [3m30s elapsed]
╷
│ Error: waiting for EC2 NAT Gateway (nat-0df5ec3ea661a14aa) create: unexpected state 'failed', wanted target 'available'. last error: Resource.AlreadyAssociated: Elastic IP address [eipalloc-0f95df00baa48f688] is already associated
│
│   with module.networking.module.vpc.aws_nat_gateway.this[0],
│   on .terraform/modules/networking.vpc/main.tf line 1080, in resource "aws_nat_gateway" "this":
│ 1080: resource "aws_nat_gateway" "this" {
│
╵

not sure if you have any idea about how to overcome this.

[pdrastil]

@darkxeno This might be related to issue in AWS provider I've bumped into. When testing the snippet I've posted I was hitting errors where provider didn't know if resource should be updated or replaced. To keep it simple I haven't proceed with the rest of implementation with providing custom IPs and left the fork for upstream so they can do the repro.

[flaviomoringa]

We really need this too, please prioritise it.
Thanks

[scamp]

We definetely need this as well.

[rajcheval]

We also need this. We added secondary IP address manually a while ago with hopes that module will support it.

[tkatrichenko]

No updates?

[flaviomoringa]

Seems some work is being done here: #1109 but has not moved for some weeks ;-(

[velkovb]

https://github.com/terraform-aws-modules/terraform-aws-vpc/pull/1109/files#r1823054641 - I did some testing and it seems this is not really supported in a nice way by the AWS provider.

hashicorp/terraform-provider-aws#34268

hashicorp/terraform-provider-aws#33964