terraform destroy for Azure Machine Learning Service ends up with an error

[galiya]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

• Terraform v0.12.24
• ARM version = "~>2.3.0"

Affected Resource(s)

• azurerm_machine_learning_workspace

Description

Creating a terraform infrastructure that provisions:

• azure resource group
• azure ml workspace, together with
  • azure key vault
  • azure storage account
  • azure application insights

Later running terraform destroy to remove infrastructure above. The command fails consistently with an error described below. Running terraform destroy second time removes all resources as expected.

Expected Behavior

All resources to be destroyed without any issues when running terraform destroy -var-file=<tfvars file>"

Actual Behavior

Error: Error deleting Machine Learning Workspace "" (Resource Group ""): machinelearningservices.WorkspacesClient#Delete: Failure responding to request: StatusCode=202 -- Original Error: autorest/azure: error response cannot be parsed: "" error: EOF

When running the terraform destroy for the second time, it reveals that keyvault, storage account and application insights resources and a parent resource group were not deleted. However, azure ml workspace resource has been removed despite an error on the 1st run.

References

• New Resource: Azure Machine Learning Service #3729 (closed)

[ArcturusZhang]

Hi @galiya thanks for this issue!
By reading the error message and status code, I suppose this should be an issue with the service itself.

One technical fact is that the terraform provider is using go SDK, and go SDK calls the REST API of the service, and the REST API of the service is described detailed here.
Let me explain something in the link:
The responses section describes the acceptable status code returned. In this case, "200" and "204" is acceptable, and all others (even "202" or "201") is not acceptable. Go SDK will return errors when received status codes that are not acceptable. And in your case, the response status code of the delete operation is "202", which is not in the acceptable status code defined in the specs, therefore go SDK throws an error. (and further, since the service may not regard 202 as an error, it may return 202 with some valid payload rather than an error, which explains why go SDK says error response cannot be parsed.

In summary, my first thought is that this should be an issue caused by change of service behavior. But could you please provide some more information about this so that I can confirm? The log around the DELETE request should be fine. You can get the log file by exporting the environment variables TF_LOG=TRACE and `TF_LOG_PATH="\path\to\your\log\file" to get the logs in a file.

[galiya]

Hey, thanks for the explanation. I've re-run the commands and attached the trimmed log file below.

log.txt

[ArcturusZhang]

Well, unfortunately this is indeed caused by some breaking changes from the service.
In your log, the response of the DELETE request is

[DEBUG] AzureRM Response for https://management.azure.com/subscriptions/SUBSCRIPTION_ID/resourceGroups/pcamvp-dev123-rg/providers/Microsoft.MachineLearningServices/workspaces/pcamvp-dev123-aml?api-version=2019-11-01: 
HTTP/2.0 202 Accepted
Cache-Control: no-cache
Date: Fri, 01 May 2020 10:50:20 GMT
Expires: -1
Location: https://management.azure.com/subscriptions/SUBSCRIPTION_ID/providers/Microsoft.MachineLearningServices/locations/eastus/workspaceOperationsStatus/5efa8409-6789-46ef-8e53-fc095288636b?api-version=2018-03-01-preview
Pragma: no-cache
Request-Context: appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Client-Request-Id: 5c126be6-c662-4994-8c4a-0bbaa058c314
X-Ms-Client-Session-Id: 
X-Ms-Correlation-Request-Id: 8c0b26d1-4d3a-2701-3a5f-f5476c235cc9
X-Ms-Ratelimit-Remaining-Subscription-Deletes: 14999
X-Ms-Request-Id: 16459393-90b8-4ffb-b765-16f21025c595
X-Ms-Routing-Request-Id: UKSOUTH:20200501T105020Z:16459393-90b8-4ffb-b765-16f21025c595
Content-Length: 0

The Location header is the proof of an async operation, it is a query URL, you can send GET request to this URL to get the current status of the deletion.
I will route this to the service team.

[ArcturusZhang]

This should have been resolved in PR #7635

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!