azuredevops_user_entitlement fails to re-add user once they are manually removed from the organisation

[OpsM0nkey]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and Azure DevOps Provider) Version

• Terraform v1.0.5
  • provider registry.terraform.io/hashicorp/azuread v2.1.0
  • provider registry.terraform.io/microsoft/azuredevops v0.1.7

Affected Resource(s)

• azuredevops_user_entitlement

Terraform Configuration Files

terraform {
  required_providers {
    azuredevops = {
      source  = "microsoft/azuredevops"
      version = ">=0.1.0"
    }
  }
}

provider "azuredevops" {
  org_service_url       = var.azdo_org_url
}

# add a user with a basic license
resource "azuredevops_user_entitlement" "add_user"{
  principal_name = "[email protected]"
  account_license_type = "express"
}

Debug Output

Link to debug output in Gist

Panic Output

N/A

Expected Behavior

The configuration is expected to reconcile the desired state - i.e. users are added/modified with each plan/apply of the terraform module and azuredevops_user_entitlements provider. If a user is manually removed from the AzDO org (but not from the configuration), the reconciliation process will identify the drift and put them back in.

Actual Behavior

Adding a user into the organisation through the azuredevops_user_entitlement resource for the first time works as expected. However, if that same user is subsequently removed from the organisation outside of the terraform process (i.e. manually through the UI), applying the same configuration fails because it can no longer find the same object ID. Running terraform plan we get a hint to the problem:

azuredevops_user_entitlement.add_user: Refreshing state... [id=821dd475-0460-6670-a61f-7a000c1b76a8]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # azuredevops_user_entitlement.add_user has been changed
  ~ resource "azuredevops_user_entitlement" "add_user" {
      ~ account_license_type = "express" -> "none"
        id                   = "821dd475-0460-6670-a61f-7a000c1b76a8"
      ~ licensing_source     = "account" -> "auto"
        # (4 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the
following plan may include actions to undo or respond to these changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following    
symbols:
  ~ update in-place

Terraform will perform the following actions:

  # azuredevops_user_entitlement.add_user will be updated in-place
  ~ resource "azuredevops_user_entitlement" "add_user" {
      ~ account_license_type = "none" -> "express"
        id                   = "821dd475-0460-6670-a61f-7a000c1b76a8"
      ~ licensing_source     = "auto" -> "account"
        # (4 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

The above is the plan output detecting the drift in configuration. It picks up the drift in license_type and licensing_source, but not the drift in the fact the principal is no longer present. As a result, it attempts to perform an update on the record instead of attempting to recreate it (i.e. destroy and add).

As a result of the terraform apply:

azuredevops_user_entitlement.add_user: Modifying... [id=821dd475-0460-6670-a61f-7a000c1b76a8]

│ Error: Updating user entitlement: (5001) Identity not found with ID 821dd475-0460-6670-a61f-7a000c1b76a8
│
│   with azuredevops_user_entitlement.add_user,
│   on main.tf line 14, in resource "azuredevops_user_entitlement" "add_user":
│   14: resource "azuredevops_user_entitlement" "add_user"{
│

What's the impact?
It makes it difficult to rely on the azdo provider to provide a reliable mechanism for reconciling desired state. In this case, where a change occurs outside of the configuration process (like manually removing a user), the configuration process will fail.

Steps to Reproduce

1. Configure your desired state (simple example in the TF configuration above, but use a real principal)
2. terraform init/plan/apply
3. Go to your AzDO org
4. Manually remove the user
5. `terraform plan/apply'
6. Get error message

Important Factoids

N/A

References

• #0000

[xuzhang3]

@OpsM0nkey It's not recommend to manage the resources out of Terraform and try handle the changes by Terraform.
Also I cannot reproduce your error, I can create the user and remove it from UI then apply terraform apply to recreate it. Normally the provider will the detect resource is no longer exists and remove it from the state.

[OpsM0nkey]

'_ It's not recommend to manage the resources out of Terraform and try handle the changes by Terraform._'
I completely agree. However, it's a likely scenario to occur, which is exactly the reason we would like to utilise Terraform to define the desired state - and reconcile in the event of divergence (however it may have happened). It's in line with one of the core principles of GitOps - Continuous reconciliation of state using automation.

With the reproduction, I wonder if this additional information will make a difference?

1. My test AzDO Org is attached to an AAD as an IdP
2. The Service Principal case matters - when the resource compares defined configuration to current state, it seems to do a case sensitive comparison. As an example, if I define '[email protected]', and the AAD service principal is '[email protected]', that difference will be picked up and the resource will trigger a replacement. Here's an example output from a `terraform plan demonstrating that point:

# azuredevops_user_entitlement.add_users["[email protected]"] must be replaced
-/+ resource "azuredevops_user_entitlement" "add_users" {
      ~ descriptor           = "aad.ODIxZGQ0NzUtMDQ2MC03NjcwLWE2MWYtN2EwMDBjMWI3NmE4" -> (known after apply)
      ~ id                   = "821dd475-0460-6670-a61f-000000" -> (known after apply)
      ~ origin               = "aad" -> (known after apply)
      ~ origin_id            = "17ccf77b-cf30-41a5-bc99-1235532" -> (known after apply)
      ~ principal_name       = "[email protected]" -> "[email protected]" # forces replacement
        # (2 unchanged attributes hidden)
    }

I hope my reply makes sense. I can replicate this error reliably every time with the code snippet I provided in the issue.

[xuzhang3]

@OpsM0nkey My test ORG connected to an AAD too, I still cannot reproduce the error. The username case sensitive issue has been fixed in #446 .
Generally, after resource has been created and managed by Terraform, when you applied terraform apply, Terraform will first perform a read operation then update, the read operation will detect the resource not longer exists and remove it from the state. In your scenario, the read operation can find the resource, but the update cannot find the resource with the same ID, have you run other parallel tasks or the user is referenced by other resource?

[OpsM0nkey]

@xuzhang3 thanks for the quick turnaround on #446!

To answer your other question, no parallel tasks run, nor is the user referenced by another resource. This is running against a clean AzDO org I just set up for testing this provider. I know what you're saying, but in my case, just running terraform plan tells me the state is only partially updated:

# azuredevops_user_entitlement.add_user has been changed
  ~ resource "azuredevops_user_entitlement" "add_user" {
      ~ account_license_type = "express" -> "none"
        id                   = "821dd475-0460-6670-a61f-7a000c1b76a8"
      ~ licensing_source     = "account" -> "auto"
        # (4 unchanged attributes hidden)
    }

i.e. it picks up on the change in license type and license source, but not id...
I can also confirm I'm running with the latest version of Terraform (1.0.6) and the azdo provider (0.1.7).

[xuzhang3]

@OpsM0nkey just recall the delete operation is an async operation and users have status . In ADO provider , we ignore the account status, this seems to be the key point here.

[xuzhang3]

close this issue, feel free to open another if you still have questions.