[Bug]: "registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element

[AlexandreGohier]

Related:

• aws_wafv2_web_acl : Provider produced inconsistent final plan #23390
• aws_wafv2_web_acl apply fails when default action changes #23423
• Error: Provider produced inconsistent final plan - invalid new value for .rule: planned set element #23936
• aws_wafv2_web_acl - Error: Provider produced inconsistent final plan #23992
• apply default tags to wafv2 or cloudfront_distribution and error #24386
• [Bug]: Error "Provider produced inconsistent final plan" when adding a default tag to a aws_wafv2_web_acl #27175
• [Bug]: Provider produced inconsistent final plan for complex aws_wafv2_web_acl configurations #27273
• [Bug]: aws_wafv2_web_acl description/tag changes result in inconsistent final plan #27479
• 👉 [Bug]: "registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element #28191
• [Bug]: aws_wafv2_web_acl is unable to use dynamic block to manage "rule_action_override" #28672
• [Bug]: Error: Provider produced inconsistent final plan #29012
• [Bug]: Error: Provider produced inconsistent final plan #29304
• [Bug]: aws_wafv2_web_acl Error: Provider produced inconsistent final plan #30858

Terraform Core Version

1.3.5

AWS Provider Version

4.45.0

Affected Resource(s)

• aws_wafv2_web_acl

Expected Behavior

We should now use rule_action_override

           dynamic rule_action_override {
              for_each = rule.value.excluded_rules
              content {
                action_to_use {
                  count {}
                }
                name = rule_action_override.value
              }
            } 

instead of deprecated excluded_rule

            dynamic "excluded_rule" {
              for_each = rule.value.excluded_rules
              content {
                name = excluded_rule.value
              }
            }

Actual Behavior

When using dynamic rule_action_override block, the webacl gets created or updated as expected.
However, subsequent updates are impossible : Error: Provider produced inconsistent final plan

Reverting to excluded_rule allows new updates to the webacl.

Relevant Error/Panic Output Snippet

Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.
  Enter a value: yes
  
Error: Provider produced inconsistent final plan

When expanding the plan for aws_wafv2_web_acl.test_alb_webacl to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element
cty.ObjectVal(map[string]cty.Value{"action":cty.ListValEmpty(cty.Object(map[string]cty.Type{"allow":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"block":cty.List(cty.Object(map[string]cty.Type{"custom_response":cty.List(cty.Object(map[string]cty.Type{"custom_response_body_key":cty.String,
"response_code":cty.Number, "response_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"captcha":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
"count":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))}))})), "name":cty.StringVal("AWSManagedRulesCommonRuleSet"),
"override_action":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"count":cty.ListValEmpty(cty.EmptyObject),
"none":cty.ListVal([]cty.Value{cty.EmptyObjectVal})})}), "priority":cty.NumberIntVal(2100),
"rule_label":cty.SetValEmpty(cty.Object(map[string]cty.Type{"name":cty.String})),
"statement":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"and_statement":cty.ListValEmpty(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"and_statement":cty.List(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"and_statement":cty.List(cty.Object(map[string]cty.Type{"statement":cty.List(cty.Object(map[string]cty.Type{"byte_match_statement":cty.List(cty.Object(map[string]cty.Type{"field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
"body":cty.List(cty.EmptyObject),
"cookies":cty.List(cty.Object(map[string]cty.Type{"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"excluded_cookies":cty.List(cty.String), "included_cookies":cty.List(cty.String)})), "match_scope":cty.String,
"oversize_handling":cty.String})),
"headers":cty.List(cty.Object(map[string]cty.Type{"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"excluded_headers":cty.List(cty.String), "included_headers":cty.List(cty.String)})), "match_scope":cty.String,
"oversize_handling":cty.String})),
"json_body":cty.List(cty.Object(map[string]cty.Type{"invalid_fallback_behavior":cty.String,
"match_pattern":cty.List(cty.Object(map[string]cty.Type{"all":cty.List(cty.EmptyObject),
"included_paths":cty.List(cty.String)})), "match_scope":cty.String, "oversize_handling":cty.String})),
"method":cty.List(cty.EmptyObject), "query_string":cty.List(cty.EmptyObject),
"single_header":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
"single_query_argument":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
"uri_path":cty.List(cty.EmptyObject)})), "positional_constraint":cty.String, "search_string":cty.String,
"text_transformation":cty.Set(cty.Object(map[string]cty.Type{"priority":cty.Number, "type":cty.String}))})),
"geo_match_statement":cty.List(cty.Object(map[string]cty.Type{"country_codes":cty.List(cty.String),
"forwarded_ip_config":cty.List(cty.Object(map[string]cty.Type{"fallback_behavior":cty.String,
"header_name":cty.String}))})), "ip_set_reference_statement":cty.List(cty.Object(map[string]cty.Type{"arn":cty.String,
"ip_set_forwarded_ip_config":cty.List(cty.Object(map[string]cty.Type{"fallback_behavior":cty.String,
"header_name":cty.String, "position":cty.String}))})),
"label_match_statement":cty.List(cty.Object(map[string]cty.Type{"key":cty.String, "scope":cty.String})),
"regex_match_statement":cty.List(cty.Object(map[string]cty.Type{"field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
"body":cty.List(cty.EmptyObject),

[...]

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform Configuration Files

resource "aws_wafv2_web_acl" "test_alb_webacl" {
  name        = "${var.test_vpc_name}"
  description = "WebACL for ${var.test_vpc_name}"
  scope       = "REGIONAL"

  default_action {
    block {}
  }

[...]

  # Managed rules
  dynamic "rule" {
    for_each = var.test_rule_groups
      content {
        name     = rule.key
        priority = rule.value.priority

        override_action {
          none {}
        }

        statement {
          managed_rule_group_statement {
            name        = rule.key
            vendor_name = rule.value.vendor_name

            dynamic rule_action_override {
               for_each = rule.value.excluded_rules
               content {
                 action_to_use {
                   count {}
                 }
    
                 name = rule_action_override.value
               }
             } 

          }
        }
        
      }
  }

[...]


}

Steps to Reproduce

1- Create or update webacl to use rule_action_override (and apply)
2- Modify something in the code that will require a webacl update (and apply) --> this will produce an inconsistant final plan
3- Revert to excluded_rule (and apply) --> works fine and further webacl updates are possible

Debug Output

No response

Panic Output

No response

Important Factoids

Tested on Ubuntu 22.04

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[javiln8]

Having the same issue here, right after start using the rule_action_override block.

[javiln8]

This is the change that introduced this new feature, is it worth reverting?
#27954

[jameslinnell]

Same issue terraform v1.1.7 aws v4.45.0
count seems to fail but allow works.
rule_action_override { action_to_use { count {} } name = "NoUserAgent_HEADER" }

rule_action_override { action_to_use { allow {} } name = "NoUserAgent_HEADER" }
originally used exclude_rule

[dp-clark1977]

same issue in our environment - reverting to the depreciated block allowed us to update the ACL successfully.

[ppshein]

that issue has been resolved? I am having the same issue as well.

Blocks of type "rule_action_override" are not expected here.

[ewbankkit]

Similar #27479.

[88lexd]

I am also facing similar issue #28672

[bclodius]

As a workaround I have been tainting my web acl when rules are impacting changes to webacl.

I keep the same webacl name so that my WAF logs do not get impacted.

Edit: See next comment below for a workaround that works for me

[bclodius]

Issue debugging

@ewbankkit I think I found something leading to part of the root cause here based on some investigation I did on the plan json that can reproduce the issue and a "workaround". Will dig more when I get a chance but this seems to be a workaround in my case.

This schema here is an optional list:

terraform-provider-aws/internal/service/wafv2/schemas.go

Line 591 in f7cf135

func customRequestHandlingSchema() *schema.Schema {

When I add or modify any aws_wafv2_web_acl.rule attribute or add a new aws_wafv2_web_acl.rule it creates the following discrepancy for all rules for aws_wafv2_web_acl.rule.statement.managed_rule_group_statement.rule_action_override.action_to_user.*.custom_request_handling

In my plan file's json the resource_changes section shows this...

# Before section...
"action_to_use": [
                            {
                              "allow": [],
                              "block": [],
                              "captcha": [],
                              "count": [
                                {
                                  "custom_request_handling": null
                                }
                              ]
                            }
                          ]

# After section...
"action_to_use": [
                            {
                              "allow": [],
                              "block": [],
                              "captcha": [],
                              "count": [
                                {
                                  "custom_request_handling": []
                                }
                              ]
                            }
                          ]

Workaround

If I ensure there is something populated in custom_request_handling then this error is fixed for my use case.

# BEFORE AND CAUSING ISSUES
dynamic "rule_action_override" {
              for_each = managed_rule_group_statement.value.count_rule_action_overrides
              content {
                action_to_use {
                  count {
                  }
                }
                name = rule_action_override.key
              }
            }

# AFTER
dynamic "rule_action_override" {
              for_each = managed_rule_group_statement.value.count_rule_action_overrides
              content {
                action_to_use {
                  count {
                  custom_request_handling {
                      insert_header {
                        name  = "TFWorkaround"
                        value = "WAF"
                      }
                    }
                  }
                }
                name = rule_action_override.key
              }
            }

Once I make these changes the before and after plans still show changes to all the rules but don't crash on apply.

[timharsch]

Thank you @bclodius for the work around!! I can confirm that changing action_to_use.count to include the custom_requrest_handling.insert-header block was able to get me past the issue on AWS v4.53.0. Seems like a fairly high severity bug since it prevents a successful terraform apply. I hope it gets fixed soon.

[blaargh]

This bug is not limited to the waf2 resource. I got the error on routing tables:

When expanding the plan for module.network.aws_route_table.<table>[1] to include new values learned so
 far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .route: planned set
 element cty.ObjectVal(map[string]cty.Value{"carrier_gateway_id":cty.NullVal(cty.String),
 "cidr_block":cty.StringVal("0.0.0.0/0"), "core_network_arn":cty.NullVal(cty.String),
 "destination_prefix_list_id":cty.NullVal(cty.String), "egress_only_gateway_id":cty.NullVal(cty.String),
 "gateway_id":cty.NullVal(cty.String), "instance_id":cty.NullVal(cty.String), "ipv6_cidr_block":cty.NullVal(cty.String),
 "local_gateway_id":cty.NullVal(cty.String), "nat_gateway_id":cty.StringVal("nat-abcd12345"),
 "network_interface_id":cty.NullVal(cty.String), "transit_gateway_id":cty.NullVal(cty.String),
 "vpc_endpoint_id":cty.NullVal(cty.String), "vpc_peering_connection_id":cty.NullVal(cty.String)}) does not correlate with
 any element in actual.

I hope this gets fixed soon, because this breaks all new applies...

[amitsamal94]

@bclodius the workaround which you mentioned what exactly does it do? Just wondering if there's some impact i need to careful about

[bclodius]

@amitsamal94 the workaround just ensures that the plan doesn't end up in a situation that causes the error. The bug gets triggered when there's NOTHING overridden in the override. In my case I had to force an extra header in the rule override config. This extra header doesn't impact my application.

[amitsamal94]

@bclodius , thanks

[dowlingw]

I can confirm this bug is present on v4.52.0 also.

[nodomain]

Please fix this with high priority. I currently breaks central WAF rule changes.

[nodomain]

Workaround helps to ease the pain. Thx so far!

[andrey-yakubovskiy]

v4.59.0 is also affected.

[olearylynda]

Also see this error. Do we know when a fix will be in place?

[k-drumz]

Is there any update on this? Confirming bug on resource tag update operations, at the very least. Terraform core version: 1.2.3, AWS provider version: v4.19.0. Thank you!

[smartpierre]

Having the same issue, Terraform 1.3.3 and AWS provider 4.63.0

[0xabdi]

Facing the same issue in terraform ~> 1.3.0 and AWS provider 4.63.0

[Hasgaroth]

I am experiencing the same issue, but not with the action_to_use override construct, but with a regular allow or count within a dynamic rule creation.

Using the same method described above, the workaround is successful here too.

  dynamic "rule" {
    for_each = var.custom_rules

    content {
      name     = rule.key
      priority = try(rule.value["priority"], "100")

      # What rule action do we want to apply
      action {
        dynamic "allow" {
          for_each = rule.value["action"] == "allow" ? [true] : []
          content {
            # --------------------------------------------------------------------
            # Adding a custom header as a workaround for a BUG in the AWS provider
            # See https://github.com/hashicorp/terraform-provider-aws/issues/28191
            # --------------------------------------------------------------------
            custom_request_handling {
              insert_header {
                name  = "TFWorkaround"
                value = "WAF"
              }
            }
          }
        }
  :
  :
  :

[bretlowery]

V5.0.1 removes excluded_rules, so rule_action_override is the only alternative. Was this fixed in 5.0.1 or is it still broken?

[YakDriver]

NOTE: I cannot reproduce this error using Terraform v1.5+/AWS provider v5.7+ after trying various configurations. Retry using a minimum of Terraform v1.4.2/AWS provider v4.67.0 but preferably Terraform v1.5.3+/AWS provider v5.8.0+ and let us know if this is still a problem! If we don't hear back and can't reproduce, we plan to close this on or around July 20, 2023. The evidence suggests this is OBE (ie, fixed in the interim).

For more details see #23992 (comment) and #28672 (comment).

[bclodius]

@YakDriver thanks for the update. Do you have any suggestions on the minimum aws provider version to try this on?

[AlexandreGohier]

With Terraform v1.5.3 and AWS provider v5.8.0, I am no longer experiencing this error! Thanks @YakDriver

[YakDriver]

@YakDriver thanks for the update. Do you have any suggestions on the minimum aws provider version to try this on?

@bclodius It depends on the exact subpart of this family of issues. If yours is similar to the op on this issue, I suggest trying a minimum of Terraform v1.4.2 and AWS provider v4.67.0.

[YakDriver]

With Terraform v1.5.3 and AWS provider v5.8.0, I am no longer experiencing this error! Thanks @YakDriver

Thanks @AlexandreGohier for reporting back!

[justinretzolk]

Hi all 👋 As was mentioned above, this issue appears to be fixed when using a minimum Terraform version of 1.4.2 and a minimum AWS Provider version of 4.67.0 (preferably Terraform 1.5.3 or later and AWS Provider 5.8.0 or later). If you experience additional unexpected behaviors with versions that meet these parameters, please open a new issue so that we can investigate further.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.