Adding security rule AFTER applying causes error

[timothykimball]

Version of terraform: v0.5.3
Steps to reproduce:

1. apply the below via terraform
2. uncomment sg_rule_https_ingress in 'rules/main.tf'
3. apply again and see the below.

Just to highlight, the second rule I added was for port 443. For some reason terraform is re-creating the port 80 rule even through that was already created in step (1)

As always, I think this is a bug - but I understand I may have made a mistake.

Result:

module.base.sg_rule_http_ingress.aws_security_group_rule.rule.0: Creating...
  cidr_blocks.#:            "" => "1"
  cidr_blocks.0:            "" => "10.0.20.1/32"
  from_port:                "" => "80"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-9b0d37fe"
  self:                     "" => "0"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "80"
  type:                     "" => "ingress"
module.base.sg_rule_http_ingress.aws_security_group_rule.rule.1: Creating...
  cidr_blocks.#:            "" => "1"
  cidr_blocks.0:            "" => "10.0.21.1/32"
  from_port:                "" => "80"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-9b0d37fe"
  self:                     "" => "0"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "80"
  type:                     "" => "ingress"
module.base.sg_rule_https_ingress.aws_security_group_rule.rule.0: Creating...
  cidr_blocks.#:            "" => "1"
  cidr_blocks.0:            "" => "10.0.20.1/32"
  from_port:                "" => "443"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-9b0d37fe"
  self:                     "" => "0"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "443"
  type:                     "" => "ingress"
module.base.sg_rule_https_ingress.aws_security_group_rule.rule.1: Creating...
  cidr_blocks.#:            "" => "1"
  cidr_blocks.0:            "" => "10.0.21.1/32"
  from_port:                "" => "443"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-9b0d37fe"
  self:                     "" => "0"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "443"
  type:                     "" => "ingress"

Error applying plan:

1 error(s) occurred:

* 2 error(s) occurred:

* 1 error(s) occurred:

* Error authorizing security group rules rules: InvalidPermission.Duplicate: the specified rule "peer: 10.0.21.1/32, TCP, from port: 80, to port: 80, ALLOW" already exists
    status code: 400, request id: []
* 1 error(s) occurred:

* Error authorizing security group rules rules: InvalidPermission.Duplicate: the specified rule "peer: 10.0.20.1/32, TCP, from port: 80, to port: 80, ALLOW" already exists
    status code: 400, request id: []

main.tf

provider "aws" {
    access_key = "${var.access_key}"
    secret_key = "${var.secret_key}"
    region = "${var.region}"
}

module "base" {
    source = "rules"
    vpc_name = "${var.vpc_name}"
    vpc_id = "${var.vpc_id}"
}

rules/main.tf

resource "aws_security_group" "nat" {
     # See http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html
     name = "${var.vpc_name}-sg_rule_bug"
     description = "Allows SSH, HTTP, and HTTPS access from internal EC2 instances through the NAT"
     vpc_id = "${var.vpc_id}"
}

module "sg_rule_http_ingress" {
    source = "../host_security_group_rule"
    port = 80
    ips = "10.0.20.1,10.0.21.1"
    sg_id = "${aws_security_group.nat.id}"
    type = "ingress"
}

/*
module "sg_rule_https_ingress" {
    source = "../host_security_group_rule"
    port = 443
    ips = "10.0.20.1,10.0.21.1"
    sg_id = "${aws_security_group.nat.id}"
    type = "ingress"
}*/


resource "aws_subnet" "subnet" {
    vpc_id = "${var.vpc_id}"
    cidr_block = "10.0.90.0/24"
    availability_zone = "eu-west-1a"
    map_public_ip_on_launch = true
    tags {
       Name = "${var.vpc_name}-sg_bug-eu-west-1a-${count.index}"
    }
    count = 1
}

resource "aws_instance" "nat" {
    ami = "ami-ef76e898"
    instance_type = "t2.micro"
    key_name = "aire-root"
    subnet_id = "${element(aws_subnet.subnet.*.id, 0)}"
    security_groups = ["${aws_security_group.nat.id}"]
    # See: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html
    source_dest_check = false
    tags {
        Name = "${var.vpc_name}-dmz-eu-west-1a-nat"
    }
}

host_security_group_rule/main.tf

resource "aws_security_group_rule" "rule" {
    type = "${var.type}"
    from_port = "${var.port}"
    to_port = "${var.port}"
    protocol = "tcp"
    cidr_blocks = ["${concat(element(split(",",var.ips), count.index), "/32")}"]
    security_group_id = "${var.sg_id}"
    count = "${length(split(",", var.ips))}"
}

[catsby]

Hey @timothykimball – what version of Terraform are you using? There was a recent bug very similar to this that I fixed and is now patched in the master branch, so I want to see if we're talking about the same thing.

[timothykimball]

terraform 0.5.3

[timothykimball]

@catsby - I have tried using Terraform v0.6.0-dev (cddd54c)

and I now receive:

• [WARN] A duplicate Security Group rule was found. This may be
  a side effect of a now-fixed Terraform issue causing two security groups with
  identical attributes but different source_security_group_ids to overwrite each
  other in the state. See Consider security groups with source security groups when hashing #2376 for more
  information and instructions for recovery. Error message: the specified rule "peer: 10.0.21.1/32, TCP, from port: 5432, to port: 5432, ALLOW" already exists

[catsby]

Hey @timothykimball

Sorry for the trouble here. Unforunately as #2376 explains, there was a bug in 0.5.x that in certain situations with Security Group Rules, the state file would not be saved correctly leading to the behavior you describe here. By upgrading, a migration is ran internally to attempt to correct this, but as you've seen there exists a scenario that it cannot mend.

The remediation is to manually identify and remove the rule in the AWS Console. In your case, you need to find the security group in question and remove the matching port 5432 rule. After doing so, re-running terraform plan and/or terraform apply should reinstate the rule and you should be all set from there.

I apologize for the trouble here. Please let me know if you need anything else.

[timothykimball]

@catsby -

Right. Did all of that. But I I have to remove the rules every time I do any kind of change to the terraform scripts, otherwise I get the following errors

* [WARN] A duplicate Security Group rule was found. This may be
a side effect of a now-fixed Terraform issue causing two security groups with
identical attributes but different source_security_group_ids to overwrite each
other in the state. See https://github.com/hashicorp/terraform/pull/2376 for more
information and instructions for recovery. Error message: the specified rule "peer: 10.0.20.10/32, TCP, from port: 443, to port: 443, ALLOW" already exists

I even blew my state files away and started from scratch on the off chance it was a lingering state bug from v0.5.3.

I am still using Terraform v0.6.0-dev (cddd54c)

[timothykimball]

@catsby - Sorry - a nob here - Do I need to re-open bug? Shall I file another one?

[catsby]

Sorry for the trouble, I'll take another look here 😦

[johnrengelman]

@catsby I'm running on the current master and aws_security_group_rule is not being saved to the state file after a successful apply. That's probably what he's running into here. I'm battling the same issue.

[catsby]

Correct. I need to refactor security group rules here. Essentially, a specific rule resource can't support multiple rules on AWS side. Specifically speaking to this issue, there are two rule resources being generated, and they only differ by the CIDR. On the AWS side, however, that's resulting in a single "rule" with 2 IPRanges in the IpPermissions as described here.

• http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AuthorizeSecurityGroupIngress.html

The rule(s) on AWS side end up looking like this:

[{
  FromPort: 80,
  IPProtocol: "tcp",
  IPRanges: [{
      CIDRIP: "10.0.20.1/32"
    },{
      CIDRIP: "10.0.21.1/32"
    }],
  ToPort: 80
}]

This is most likely due to Terraform using the IPPermissions field there, instead of the top-level CIDRIP. This is a carry-over from the Security Group resource when I created this resource, and in hindsight doesn't map to individual rules like I had hoped.

The refactoring will likely limit each security group rule to only support a single cidr block. Still digging in though, we'll see. Sorry for all the trouble here 😢

[catsby]

@timothykimball – as a stopgap, you could remove the count on your security group rule, and change the cidr_blocks , as shown:

resource "aws_security_group_rule" "rule" {
    type = "ingress"
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["${formatlist("%s/32", split(",",var.ips))}"]
    security_group_id = "${aws_security_group.bastion.id}"
}

That should create a single rule with multiple cidr_blocks entries for the var.ips you have defined. As I mentioned, I plan on refactoring the security group rules, and in the future your current syntax will work, given you specify a single cidr_block . I can update here when that pull request is ready, but it may take some time. Sorry again for the hassle

[johnrengelman]

@catsby just wondering, I'm trying to add a rule to a security group that is managed externally. Is this going to work? I don't see terraform store anything in the state file for aws_security_group_rule

[timothykimball]

@catsby - Work around is working :D

[timothykimball]

@catsby - Work around has stopped working :(

module.base.bastion.ingress_bastion_ssh.aws_security_group_rule.rule: Creating...
  cidr_blocks.#:            "" => "2"
  cidr_blocks.0:            "" => "10.0.10.10/32"
  cidr_blocks.1:            "" => "10.0.11.10/32"
  from_port:                "" => "22"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-8f1316ea"
  self:                     "" => "0"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "22"
  type:                     "" => "ingress"
module.base.bastion.ingress_bastion_ssh.aws_security_group_rule.rule: Error: 1 error(s) occurred:
* [WARN] A duplicate Security Group rule was found. This may be
a side effect of a now-fixed Terraform issue causing two security groups with
identical attributes but different source_security_group_ids to overwrite each
other in the state. See https://github.com/hashicorp/terraform/pull/2376 for more
information and instructions for recovery. Error message: the specified rule "peer: 10.0.10.10/32, TCP, from port: 22, to port: 22, ALLOW" already exists

Is consistently appearing when using the following rules

resource "aws_security_group" "bastion" {
     name = "${var.vpc_name}-bastion"
     description = "Allows SSH access from VPN servers. Allows SSH access from bastion to VPN, NAT, and App servers."
     vpc_id = "${var.vpc_id}"
}

module "sg_rule_http_egress" {
    source = "../../cidr_security_group_rule"
    port = 80
    protocol = "tcp"
    cidr_blocks = "0.0.0.0/0"
    sg_id = "${aws_security_group.bastion.id}"
    type = "egress"
}

module "sg_rule_https_egress" {
    source = "../../cidr_security_group_rule"
    port = 443
    protocol = "tcp"
    cidr_blocks = "0.0.0.0/0"
    sg_id = "${aws_security_group.bastion.id}"
    type = "egress"
}

module "egress_bastion_ssh" {
    source = "../../host_security_group_rule"
    port = 22
    ips = "${var.ssh_ips_egress}"
    sg_id = "${aws_security_group.bastion.id}"
    type = "egress"
}

module "ingress_bastion_ssh" {
    source = "../../host_security_group_rule"
    port = 22
    ips = "${var.ssh_ips_ingress}"
    sg_id = "${aws_security_group.bastion.id}"
    type = "ingress"
}

where host_security_group_rule is:

resource "aws_security_group_rule" "rule" {
    type = "${var.type}"
    from_port = "${var.port}"
    to_port = "${var.port}"
    protocol = "${var.protocol}"
    cidr_blocks = ["${formatlist("%s/32", split(",",var.ips))}"]
    security_group_id = "${var.sg_id}"
}

[catsby]

Hey @timothykimball – thanks for the detailed information. Unfortunately, I'm having trouble reconstructing your example. Is there a variable.tf or other that I'm missing?

[catsby]

I just opened #3019 to fix this, though being unable to directly reconstruct your example, I don't know for sure. If possible please check it out!

[catsby]

Hey all – I'm going to close this issue for now, due to #3019 being merged and general inactivity. If you're still experience this, please let us know.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.