Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: aws_wafv2_web_acl is unable to use dynamic block to manage "rule_action_override" #28672

Closed
88lexd opened this issue Jan 5, 2023 · 11 comments
Assignees
Labels
bug Addresses a defect in current functionality. service/wafv2 Issues and PRs that pertain to the wafv2 service.

Comments

@88lexd
Copy link

88lexd commented Jan 5, 2023

Related:

Terraform Core Version

1.3.0

AWS Provider Version

4.44.0

Affected Resource(s)

aws_wafv2_web_acl

Expected Behavior

Certain rule actions I want to set to count mode should work via my dynamic block

Actual Behavior

Terraform errors out and ends with "This is a bug in the provider, which should be reported in the provider's own issue tracker"

Relevant Error/Panic Output Snippet

aws_wafv2_web_acl.regional_waf: Refreshing state... [id=bdb18064-f4b7-4484-ba1b-bfef65760385]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_wafv2_web_acl.regional_waf will be updated in-place
  ~ resource "aws_wafv2_web_acl" "regional_waf" {
        id         = "bdb18064-f4b7-4484-ba1b-bfef65760385"
        name       = "reproduce_bug_test"
        tags       = {}
        # (5 unchanged attributes hidden)

      + rule {
          + name     = "AWSManagedRulesAmazonIpReputationList"
          + priority = 2

          + override_action {

              + none {}
            }

          + statement {

              + managed_rule_group_statement {
                  + name        = "AWSManagedRulesAmazonIpReputationList"
                  + vendor_name = "AWS"

                  + rule_action_override {
                      + name = "AWSManagedIPReputationList"

                      + action_to_use {

                          + count {
                            }
                        }
                    }
                  + rule_action_override {
                      + name = "AWSManagedReconnaissanceList"

                      + action_to_use {

                          + count {
                            }
                        }
                    }
                }
            }

          + visibility_config {
              + cloudwatch_metrics_enabled = false
              + metric_name                = "AWSManagedRulesAmazonIpReputationList"
              + sampled_requests_enabled   = false
            }
        }
      - rule {
          - name     = "AWSManagedRulesKnownBadInputsRuleSet" -> null
          - priority = 1 -> null

          - override_action {

              - none {}
            }

          - statement {

              - managed_rule_group_statement {
                  - name        = "AWSManagedRulesKnownBadInputsRuleSet" -> null
                  - vendor_name = "AWS" -> null

                  - rule_action_override {
                      - name = "Log4JRCE_QUERYSTRING" -> null

                      - action_to_use {

                          - count {
                            }
                        }
                    }
                }
            }

          - visibility_config {
              - cloudwatch_metrics_enabled = false -> null
              - metric_name                = "AWSManagedRulesKnownBadInputsRuleSet" -> null
              - sampled_requests_enabled   = false -> null
            }
        }
      + rule {
          + name     = "AWSManagedRulesKnownBadInputsRuleSet"
          + priority = 1

          + override_action {

              + none {}
            }

          + statement {

              + managed_rule_group_statement {
                  + name        = "AWSManagedRulesKnownBadInputsRuleSet"
                  + vendor_name = "AWS"

                  + rule_action_override {
                      + name = "Log4JRCE_QUERYSTRING"

                      + action_to_use {

                          + count {
                            }
                        }
                    }
                }
            }

          + visibility_config {
              + cloudwatch_metrics_enabled = false
              + metric_name                = "AWSManagedRulesKnownBadInputsRuleSet"
              + sampled_requests_enabled   = false
            }
        }

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Error: Provider produced inconsistent final plan

When expanding the plan for aws_wafv2_web_acl.regional_waf to include new
values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.rule: planned set element
cty.ObjectVal(map[string]cty.Value{"action":cty.ListValEmpty(cty.Object(map[string]cty.Type{"allow":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"block":cty.List(cty.Object(map[string]cty.Type{"custom_response":cty.List(cty.Object(map[string]cty.Type{"custom_response_body_key":cty.String,
...
"visibility_config":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cloudwatch_metrics_enabled":cty.False,
"metric_name":cty.StringVal("AWSManagedRulesKnownBadInputsRuleSet"),
"sampled_requests_enabled":cty.False})})}) does not correlate with any
element in actual.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Terraform Configuration Files

This is the short version of the Terraform code I have

terraform {
  required_version = ">= 1.3"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.44.0"
    }
  }
}

provider "aws" {
  alias  = "regional"
  region = "ap-southeast-2"
}

variable "rulesets" {
  # Using "type any" here for simplicity to reproduce bug
  type = any
  default = [
    {
      rule_name               = "AWSManagedRulesKnownBadInputsRuleSet"
      priority                = 1
      rules_override_to_count = ["Log4JRCE_QUERYSTRING"]
    },
    # {
    #   rule_name               = "AWSManagedRulesAmazonIpReputationList"
    #   priority                = 2
    #   rules_override_to_count = ["AWSManagedIPReputationList", "AWSManagedReconnaissanceList"]
    # }
  ]
}

resource "aws_wafv2_web_acl" "regional_waf" {
  scope = "REGIONAL"

  name        = "reproduce_bug_test"

  default_action {
    allow {}
  }

  #######################
  # Begin Rules
  dynamic "rule" {
    for_each = { for this in var.rulesets : this.rule_name => this }
    content {
      name     = rule.key
      priority = rule.value.priority

      override_action {
        none {}
      }

      statement {
        managed_rule_group_statement {
          name        = rule.key
          vendor_name = "AWS"

          dynamic "rule_action_override" {
            for_each = [for rule_override in rule.value.rules_override_to_count : rule_override]

            content {
              name = rule_action_override.value
              action_to_use {
                count {}
              }
            }
          }
        }
      }

      visibility_config {
        cloudwatch_metrics_enabled = false
        metric_name                = rule.key
        sampled_requests_enabled   = false
      }
    }
  }
  #######################
  # End Rules

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "test-waf-metrics"
    sampled_requests_enabled   = true
  }
}

Steps to Reproduce

When I uncomment the variable from my configuration file so the variable looks like this, then Terraform errors out

variable "rulesets" {
  # Using "type any" here for simplicity to reproduce bug
  type = any
  default = [
    {
      rule_name               = "AWSManagedRulesKnownBadInputsRuleSet"
      priority                = 1
      rules_override_to_count = ["Log4JRCE_QUERYSTRING"]
    },
    {
      rule_name               = "AWSManagedRulesAmazonIpReputationList"
      priority                = 2
      rules_override_to_count = ["AWSManagedIPReputationList", "AWSManagedReconnaissanceList"]
    }
  ]
}

I notice the bug only happens if I am using "rules_override_to_count". Say for example if I have the following variables, I can add/remove as many rules as I want. As long as I am not using the dynamic "rule_action_override" block.

variable "rulesets" {
  # Using "type any" here for simplicity to reproduce bug
  type = any
  default = [
    {
      rule_name               = "AWSManagedRulesKnownBadInputsRuleSet"
      priority                = 1
      rules_override_to_count = []
    },
    {
      rule_name               = "AWSManagedRulesAmazonIpReputationList"
      priority                = 2
      rules_override_to_count = []
    },
    {
      rule_name               = "AWSManagedRulesBotControlRuleSet"
      priority                = 3
      rules_override_to_count = []
    }
  ]
}

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

@88lexd 88lexd added bug Addresses a defect in current functionality. needs-triage Waiting for first response or review from a maintainer. labels Jan 5, 2023
@github-actions
Copy link

github-actions bot commented Jan 5, 2023

Community Note

Voting for Prioritization

  • Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

  • If you are interested in working on this issue, please leave a comment.
  • If this would be your first contribution, please review the contribution guide.

@github-actions github-actions bot added the service/wafv2 Issues and PRs that pertain to the wafv2 service. label Jan 5, 2023
@justinretzolk justinretzolk removed the needs-triage Waiting for first response or review from a maintainer. label Jan 18, 2023
@azt3k
Copy link

azt3k commented Mar 2, 2023

I have found I also have this issue when adding or removing managed rule sets in an existing ACL. A good indication that it's going to die with Error: Provider produced inconsistent final plan seems to be that it will show rules that need to be both removed and re-added again even though those rules haven't changed.

The plan output below results from adding AWSManagedRulesAnonymousIpList to an ACL that had AWSManagedRulesAmazonIpReputationList, AWSManagedRulesATPRuleSet and AWSManagedRulesBotControlRuleSet.

Other observations I have made are;

  1. The sequence they are declared doesn't seem to matter.
  2. The same thing happens if I add/remove any manged rule(s) in an existing ACL that already contains managed rules.
  3. If I remove all managed rules and re-add what I want to in one go it works.

Plan output after adding AWSManagedRulesAnonymousIpList;

  # aws_wafv2_web_acl.web_waf will be updated in-place
  ~ resource "aws_wafv2_web_acl" "web_waf" {
        id          = "028de6e7-0a08-4b34-b8af-e027794232eb"
        name        = "waf_test_web_waf"
        tags        = {
            "Environment" = "waf-test"
            "Project"     = "waf"
        }
        # (6 unchanged attributes hidden)

      - rule {
          - name     = "AWS-AWSManagedRulesATPRuleSet-waf-test" -> null
          - priority = 35 -> null

          - override_action {

              - none {}
            }

          - statement {

              - managed_rule_group_statement {
                  - name        = "AWSManagedRulesATPRuleSet" -> null
                  - vendor_name = "AWS" -> null

                  - managed_rule_group_configs {
                      - login_path = "/Security/login" -> null
                    }
                  - managed_rule_group_configs {
                      - payload_type = "FORM_ENCODED" -> null
                    }
                  - managed_rule_group_configs {

                      - username_field {
                          - identifier = "Email" -> null
                        }
                    }
                  - managed_rule_group_configs {

                      - password_field {
                          - identifier = "Password" -> null
                        }
                    }
                }
            }

          - visibility_config {
              - cloudwatch_metrics_enabled = true -> null
              - metric_name                = "AWS-AWSManagedRulesATPRuleSet-waf-test" -> null
              - sampled_requests_enabled   = true -> null
            }
        }
      + rule {
          + name     = "AWS-AWSManagedRulesATPRuleSet-waf-test"
          + priority = 35

          + override_action {

              + none {}
            }

          + statement {

              + managed_rule_group_statement {
                  + name        = "AWSManagedRulesATPRuleSet"
                  + vendor_name = "AWS"

                  + managed_rule_group_configs {
                      + login_path = "/Security/login"
                    }
                  + managed_rule_group_configs {
                      + payload_type = "FORM_ENCODED"
                    }
                  + managed_rule_group_configs {

                      + username_field {
                          + identifier = "Email"
                        }
                    }
                  + managed_rule_group_configs {

                      + password_field {
                          + identifier = "Password"
                        }
                    }
                }
            }

          + visibility_config {
              + cloudwatch_metrics_enabled = true
              + metric_name                = "AWS-AWSManagedRulesATPRuleSet-waf-test"
              + sampled_requests_enabled   = true
            }
        }
      - rule {
          - name     = "AWS-AWSManagedRulesAmazonIpReputationList-waf-test" -> null
          - priority = 32 -> null

          - override_action {

              - none {}
            }

          - statement {

              - managed_rule_group_statement {
                  - name        = "AWSManagedRulesAmazonIpReputationList" -> null
                  - vendor_name = "AWS" -> null
                }
            }

          - visibility_config {
              - cloudwatch_metrics_enabled = true -> null
              - metric_name                = "AWS-AWSManagedRulesAmazonIpReputationList-waf-test" -> null
              - sampled_requests_enabled   = true -> null
            }
        }
      + rule {
          + name     = "AWS-AWSManagedRulesAmazonIpReputationList-waf-test"
          + priority = 32

          + override_action {

              + none {}
            }

          + statement {

              + managed_rule_group_statement {
                  + name        = "AWSManagedRulesAmazonIpReputationList"
                  + vendor_name = "AWS"
                }
            }

          + visibility_config {
              + cloudwatch_metrics_enabled = true
              + metric_name                = "AWS-AWSManagedRulesAmazonIpReputationList-waf-test"
              + sampled_requests_enabled   = true
            }
        }
      + rule {
          + name     = "AWS-AWSManagedRulesAnonymousIpList-waf-test"
          + priority = 33

          + override_action {

              + none {}
            }

          + statement {

              + managed_rule_group_statement {
                  + name        = "AWSManagedRulesAnonymousIpList"
                  + vendor_name = "AWS"

                  + rule_action_override {
                      + name = "HostingProviderIPList"

                      + action_to_use {

                          + count {
                            }
                        }
                    }
                }
            }

          + visibility_config {
              + cloudwatch_metrics_enabled = true
              + metric_name                = "AWS-AWSManagedRulesAnonymousIpList-waf-test"
              + sampled_requests_enabled   = true
            }
        }
      - rule {
          - name     = "AWS-AWSManagedRulesBotControlRuleSet-waf-test" -> null
          - priority = 36 -> null

          - override_action {

              - none {}
            }

          - statement {

              - managed_rule_group_statement {
                  - name        = "AWSManagedRulesBotControlRuleSet" -> null
                  - vendor_name = "AWS" -> null

                  - rule_action_override {
                      - name = "CategoryContentFetcher" -> null

                      - action_to_use {

                          - count {
                            }
                        }
                    }
                  - rule_action_override {
                      - name = "SignalAutomatedBrowser" -> null

                      - action_to_use {

                          - count {
                            }
                        }
                    }
                  - rule_action_override {
                      - name = "SignalNonBrowserUserAgent" -> null

                      - action_to_use {

                          - count {
                            }
                        }
                    }
                }
            }

          - visibility_config {
              - cloudwatch_metrics_enabled = true -> null
              - metric_name                = "AWS-AWSManagedRulesBotControlRuleSet-waf-test" -> null
              - sampled_requests_enabled   = true -> null
            }
        }
      + rule {
          + name     = "AWS-AWSManagedRulesBotControlRuleSet-waf-test"
          + priority = 36

          + override_action {

              + none {}
            }

          + statement {

              + managed_rule_group_statement {
                  + name        = "AWSManagedRulesBotControlRuleSet"
                  + vendor_name = "AWS"

                  + rule_action_override {
                      + name = "CategoryContentFetcher"

                      + action_to_use {

                          + count {
                            }
                        }
                    }
                  + rule_action_override {
                      + name = "SignalAutomatedBrowser"

                      + action_to_use {

                          + count {
                            }
                        }
                    }
                  + rule_action_override {
                      + name = "SignalNonBrowserUserAgent"

                      + action_to_use {

                          + count {
                            }
                        }
                    }
                }
            }

          + visibility_config {
              + cloudwatch_metrics_enabled = true
              + metric_name                = "AWS-AWSManagedRulesBotControlRuleSet-waf-test"
              + sampled_requests_enabled   = true
            }
        }

        # (14 unchanged blocks hidden)
    }

As above, when it show rules that need to be both removed and re-added again it seems to result in the same outcome;

│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for aws_wafv2_web_acl.web_waf to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element

Version info:

  • Terraform v1.3.9 on darwin_arm64
  • provider registry.terraform.io/hashicorp/aws v4.56.0

@hypsrizzari
Copy link

There are several bugs opened that look related to this issue. I was in the same boat having to taint the existing resource to get it to apply. The comment in issue 28191 resolved it for me with AWS provider 4.5x. Hope it helps. Kudos to the author.

#28191 (comment)

@daniel-brown-improving
Copy link

Did this ever get resolved? I still can't create a dynamic block with "rule_action_override", and I'm using AWS Provider 4.58.0.

@zamirTo1
Copy link

No :(

@daniel-brown-improving
Copy link

I got my issue resolved with a little help from a guy on Reddit. Here's a great example using a dynamic block with "rule_action_override". Thank you Trussworks. I'm using AWS Provider 5.4.0 and Terraform 1.50.

Check this out:
https://github.com/trussworks/terraform-aws-wafv2/blob/main/main.tf#L47-L62

@YakDriver
Copy link
Member

YakDriver commented Jul 13, 2023

NOTE: I cannot reproduce this error using Terraform v1.5+/AWS provider v5.7+ after trying various configurations. Retry using a minimum of Terraform v1.4.2/AWS provider v4.67.0 but preferably Terraform v1.5.3+/AWS provider v5.8.0+ and let us know if this is still a problem! If we don't hear back and can't reproduce, we plan to close this on or around July 20, 2023. The evidence suggests this is OBE (ie, fixed in the interim).

For more details see #23992 (comment) and #28672 (comment).

@YakDriver YakDriver self-assigned this Jul 13, 2023
@YakDriver YakDriver added the waiting-response Maintainers are waiting on response from community or contributor. label Jul 13, 2023
@88lexd
Copy link
Author

88lexd commented Jul 14, 2023

I am still facing the same issue by using my test code above to reproduce this issue. I am using the aws provider v5.8.0

$ terraform version
Terraform v1.3.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.8.0

@github-actions github-actions bot removed the waiting-response Maintainers are waiting on response from community or contributor. label Jul 14, 2023
@YakDriver
Copy link
Member

Unfortunately, I still cannot reproduce the bug. See exactly what I did below and let me know if I've missed something.

Terraform v1.4+

I believe this is fixed in Terraform v1.4+. Can you try on Terraform 1.4+?

Terraform v1.5.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.7.0

Step 1

apply this:

variable "rulesets" {
  type = any
  default = [
    {
      rule_name               = "AWSManagedRulesKnownBadInputsRuleSet"
      priority                = 1
      rules_override_to_count = ["Log4JRCE_QUERYSTRING"]
    }
  ]
}

resource "aws_wafv2_web_acl" "test" {
  scope = "REGIONAL"
  name  = "issue28672"

  default_action {
    allow {}
  }

  #######################
  # Begin Rules
  dynamic "rule" {
    for_each = { for this in var.rulesets : this.rule_name => this }
    content {
      name     = rule.key
      priority = rule.value.priority

      override_action {
        none {}
      }

      statement {
        managed_rule_group_statement {
          name        = rule.key
          vendor_name = "AWS"

          dynamic "rule_action_override" {
            for_each = [for rule_override in rule.value.rules_override_to_count : rule_override]

            content {
              name = rule_action_override.value
              action_to_use {
                count {}
              }
            }
          }
        }
      }

      visibility_config {
        cloudwatch_metrics_enabled = false
        metric_name                = rule.key
        sampled_requests_enabled   = false
      }
    }
  }
  #######################
  # End Rules

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "test-waf-metrics"
    sampled_requests_enabled   = true
  }
}

Step 2

Reapply same config without changes. No issues.

Step 3

Apply this:

variable "rulesets" {
  # Using "type any" here for simplicity to reproduce bug
  type = any
  default = [
    {
      rule_name               = "AWSManagedRulesKnownBadInputsRuleSet"
      priority                = 1
      rules_override_to_count = ["Log4JRCE_QUERYSTRING"]
    },
    {
      rule_name               = "AWSManagedRulesAmazonIpReputationList"
      priority                = 2
      rules_override_to_count = ["AWSManagedIPReputationList", "AWSManagedReconnaissanceList"]
    }
  ]
}

resource "aws_wafv2_web_acl" "test" {
  scope = "REGIONAL"
  name  = "issue28672"

  default_action {
    allow {}
  }

  #######################
  # Begin Rules
  dynamic "rule" {
    for_each = { for this in var.rulesets : this.rule_name => this }
    content {
      name     = rule.key
      priority = rule.value.priority

      override_action {
        none {}
      }

      statement {
        managed_rule_group_statement {
          name        = rule.key
          vendor_name = "AWS"

          dynamic "rule_action_override" {
            for_each = [for rule_override in rule.value.rules_override_to_count : rule_override]

            content {
              name = rule_action_override.value
              action_to_use {
                count {}
              }
            }
          }
        }
      }

      visibility_config {
        cloudwatch_metrics_enabled = false
        metric_name                = rule.key
        sampled_requests_enabled   = false
      }
    }
  }
  #######################
  # End Rules

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "test-waf-metrics"
    sampled_requests_enabled   = true
  }
}

Step 4

Apply Step 3 config again. No issues.

Step 5

Apply original Step 1 config again. No issues.

Step 6

Reapply original Step 1 config again. No issues.

@YakDriver YakDriver added the waiting-response Maintainers are waiting on response from community or contributor. label Jul 17, 2023
@88lexd
Copy link
Author

88lexd commented Jul 18, 2023

Interesting! Works for me using the following versions! will close this issue off thanks!

Terraform v1.5.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.8.0

@88lexd 88lexd closed this as completed Jul 18, 2023
@github-actions github-actions bot removed the waiting-response Maintainers are waiting on response from community or contributor. label Jul 18, 2023
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 17, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/wafv2 Issues and PRs that pertain to the wafv2 service.
Projects
None yet
Development

No branches or pull requests

7 participants