Error deleting Role Definition - Service returned an error. Status=403 Code="AuthorizationFailed"

[kaiaschulz]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.16

Affected Resource(s)

• azurerm_role_definition

Terraform Configuration Files

terraform {
  required_version = ">= 0.12.16"
}

variable "client_secret" {
  type        = string
  description = "The service principal client secred for the azurerm/ad client context."
  default     = null
}
variable "client_id" {
  type        = string
  description = "The service principal client id for the azurerm/ad client context."
  default     = null
}
variable "tenant_id" {
  type        = string
  description = "The default tenant for the azurerm client/ad context."
  default     = null
}
variable "subscription_id" {
  type        = string
  description = "The default subscription for the azurerm client context."
  default     = null
}
variable "deploymentStage" {
  type        = string
  description = "dev, qa, prod"
  default     = null
}

data "azurerm_client_config" "current" {
}

provider "azurerm" {
  version         = "~> 1.36.0"
  client_id       = var.client_id
  subscription_id = var.subscription_id
  tenant_id       = var.tenant_id
  client_secret   = var.client_secret
}
variable "scope_XXXX_AG_con_qa" {
  type        = string
  description = "This variable will be used to define the scope and assignable scope for the 'XXXX AG con_QA'"
  default     = "/providers/Microsoft.Management/managementgroups/TEST"
}

resource "azurerm_role_definition" "XXXX_AG_con_QA" {
  name        = "XXXX AG con_QA"
  scope       = var.scope_XXXX_AG_con_qa
  description = "XXXX custom role"

  permissions {
    actions     = ["Microsoft.Network/ApplicationGateways/delete",
                    "Microsoft.Network/ApplicationGateways/read",
                    "Microsoft.Network/ApplicationGateways/start/action",
                    "Microsoft.Network/ApplicationGateways/stop/action",
                    "Microsoft.Network/ApplicationGateways/write"]
    not_actions = []
    data_actions = []
    not_data_actions = []
  }

  assignable_scopes = [
    var.scope_XXXX_AG_con_qa, 
  ]
}

Debug Output

Error: Error deleting Role Definition "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" at Scope "": authorization.RoleDefinitionsClient#Delete: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY' with object id 'YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY' does not have authorization to perform action 'Microsoft.Authorization/roleDefinitions/delete' over scope '/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX' or the scope is invalid. If access was recently granted, please refresh your credentials."

Panic Output

Expected Behavior

A defined and existing custom role definition should be deleted with the terraform apply-command. Creation and modification is working as expected.

Actual Behavior

Defined roled should be deleted. After the terraform apply-command I got the following error code:
Error: Error deleting Role Definition "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" at Scope "": authorization.RoleDefinitionsClient#Delete: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY' with object id 'YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY' does not have authorization to perform action 'Microsoft.Authorization/roleDefinitions/delete' over scope '/providers/Microsoft.Authorization/roleDefinitions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX' or the scope is invalid. If access was recently granted, please refresh your credentials."

We also gave the serviceprincipal account the Owner rights on the subscription without any success.

Steps to Reproduce

1. Create a service principal
2. Grant service principal "API permissions" > "Microsoft Graph": "User.Read"
3. Create a new subscription
4. Assign the service principal the "Owner"-role on this subscription
5. Create an custome role "C1" on this subscription with terraform
6. Change the .tf-file and try to create "C2" instead

C1 should be deleted and C2 should be created
Creation will be successfull and deletion will fail.

Important Factoids

References

• #0000

[tombuildsstuff]

hi @kaiaschulz

Thanks for opening this issue - apologies for the delayed response here!

Taking a look through here unfortunately Role Assignments and Role Definitions are eventually consistent - meaning that whilst changes may show up right away on one API call, they may not in another. #9850 will handle the eventually consistent Role Definitions when updating these within Terraform, in this instance unfortunately Terraform can't wait for the Role Definition to become stable during provider initialization (since we can't guarantee if the credentials are/will ever become valid).

As such whilst I'd like to thank you for opening this issue, since the Azure API is eventually consistent here - unfortunately the workaround would be to wait for these Role Assignments/Definitions to settle down/finish replicating before calling Terraform from an external script, which should take around 5m. Whilst I appreciate that's not an ideal answer, unfortunately this is a bug in the Azure API. As mentioned above #9850 will handle the eventual consistency within the Update method for Role Definitions - as such I'm going to close this issue for the moment.

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!