azurerm: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

[MykolaPelyp-SoftServe]

Terraform (and AzureRM Provider) Version

Hi, we having that on:
azurerm version = "=2.37.0"
version = "=2.41.0"
version = "=2.30.0"

Terraform Version

terraform is written in Terraform 12 features

v13.5

Affected Resource(s)

azurerm_storage_share

Terraform Configuration Files

resource "azurerm_storage_share" "storage_share" {
  count                = var.file_share_enabled ? 1 : 0
  name                 = var.file_share_name
  storage_account_name = element(azurerm_storage_account.storage_account.*.name, 0)
  quota                = 2048

}

Debug Output

2020/12/29 18:21:53 [TRACE] dag/walk: vertex "module.storage_account (close)" is waiting for "module.storage_account.output.file_share_id (expand)"
2020/12/29 18:21:53 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.storage_account.azurerm_storage_share.storage_share (expand)"
2020/12/29 18:21:53 [TRACE] dag/walk: vertex "root" is waiting for "module.storage_account (close)"
2020/12/29 18:21:55 [TRACE] dag/walk: vertex "module.storage_account.output.file_share_id (expand)" is waiting for "module.storage_account.azurerm_storage_share.storage_share (expand)"
2020/12/29 18:21:58 [TRACE] dag/walk: vertex "module.storage_account (close)" is waiting for "module.storage_account.output.file_share_id (expand)"
2020/12/29 18:21:58 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.storage_account.azurerm_storage_share.storage_share (expand)"
2020/12/29 18:21:58 [TRACE] dag/walk: vertex "root" is waiting for "module.storage_account (close)"
2020/12/29 18:22:00 [TRACE] dag/walk: vertex "module.storage_account.output.file_share_id (expand)" is waiting for "module.storage_account.azurerm_storage_share.storage_share (expand)"
2020/12/29 18:22:03 [ERROR] eval: *terraform.EvalRefresh, err: Error retrieving File Share "thnxthnxfs" (Account "thnxthnxsa" / Resource Group "thnxthnx-rg"): shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
2020/12/29 18:22:03 [ERROR] eval: *terraform.EvalSequence, err: Error retrieving File Share "thnxthnxfs" (Account "thnxthnxsa" / Resource Group "thnxthnx-rg"): shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

Expected Behaviour

We expect our state refresh successfully, it is happening on Plan stage, first apply goes without any issues, at the moment we are using -refresh=false as a workaround

Actual Behaviour

Error: Error retrieving File Share "fthnxthnxfs" (Account "fthnxthnxsa" / Resource Group "fthnxthnx-rg"): shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

After 10 minutes of refreshing of the state,
We also have 4 other Private Endpoints (after we added them refresh took from ~30 seconds to 5 minutes, after we added fifth Private endpoint for File Share it stopped refreshing and always failing at 10 minutes)

Steps to Reproduce

resource "azurerm_storage_account" "storage_account" {
  count                     = var.enabled ? 1 : 0
  name                      = var.storage_account_name
  resource_group_name       = var.resource_group_name
  location                  = var.location
  account_kind              = var.account_kind
  account_tier              = var.account_tier
  account_replication_type  = var.account_replication_type
  enable_https_traffic_only = var.enable_https_traffic_only
  access_tier               = var.access_tier

  identity {
    type = "SystemAssigned"
  }

  network_rules {
    default_action             = var.network_rules_default_action
    bypass                     = var.network_rules_bypass
    ip_rules                   = var.network_rules_allowed_ip_ranges
    virtual_network_subnet_ids = var.network_rules_allowed_subnet_ids
  }


  tags = merge(
    var.tags,
    {
      "Name" = var.storage_account_name
    },
  )
}
resource "azurerm_storage_share" "storage_share" {
  count                = var.file_share_enabled ? 1 : 0
  name                 = var.file_share_name
  storage_account_name = element(azurerm_storage_account.storage_account.*.name, 0)
  quota                = 2048

} 

module "private_endpoint_sa_file_share" {
    source = "../module-azure-private-endpoint"
    name                            = "thnxthnx"
    location                        = var.location
    resource_group_name             = var.cluster_app_rg
    subnet_id                       = var.cluster_app_subnet_id
    private_endpoint_resource_id    = element(azurerm_storage_account.storage_account.*.id, 0)
    subresource_names               = ["file"]
    private_endpoint_enabled        = var.private_endpoint_enabled
    private_dns_zone_ids            = [module.private_dns_zone_storage_account_file_share.id]
    private_dns_name                = module.private_dns_zone_storage_account_file_share.name
}

module "private_dns_zone_storage_account_file_share" {
    source = "../module-azure-private-dns-zone"
    env                             = var.env
    dns_zone_name                   = "privatelink.file.core.windows.net"
    resource_group_name             = var.cluster_app_rg
    cluster_app_network_id          = var.cluster_app_network_id
}

1. terraform apply

Notes:
On azurerm providers 2.37 - 2.41 issue description not explaining anything:

Error: shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request

[MykolaPelyp-SoftServe]

Hi again,

I am reusing the exact same code on different environment,
There is hard coded version of AzureRM provider =2.30.0 (the same AzureRM provider as on errored env. as we tested most of the code on it)

There is no refresh issue, furthermore, terraform refresh happening in lightning speed without any issues, so main differences between errored environment and working one:

1. Terraform v0.14.7 (compared to v13.5 on errored)
2. As for now we have no firewall restrictions established on Storage Account where we store state file

[kiazhi]

Error: Error reading queue properties for AzureRM Storage Account "storageaccountname": queues.Client#GetServiceProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

• OS: RHEL - Having the issue above

  • terraform v0.14.7
  • registry.terraform.io/hashicorp/azurerm v2.50.0

• OS: macOS - No issue

  • terraform v0.14.7
  • registry.terraform.io/hashicorp/azurerm v2.50.0

[jrfoster0182]

Receiving the same issue.

OS: RHEL

• terraform v0.14.7
• registry.terraform.io/hashicorp/azurerm v2.36.0

[Chambras]

I am having this issue as well.
OS: 18.04.5 LTS (Bionic Beaver)

• terraform v0.15.4
• registry.terraform.io/hashicorp/azurerm v2.60.0

I found out it happens when you add an azurerm_private_endpoint resource for your fileshare.

[kevinsorenson]

Here too, also on private endpoint for blob storage
OS: Win 10
terraform v0.14.4
registry.terraform.io/hashicorp/azurerm v2.60.0

[RaceFPV]

seeing the same here, trying to refresh state on azurerm_storage_container fails
OS: ubuntu
terraform: v0.154
azurerm: 2.61.0

[jhughesjha]

Having the same issue
OS: Ubuntu
terraform: v1.0.0
azurerm: 2.65.0

[fgarcia-cnb]

same issue, with azurerm 2.59, rhel and private endpoint. we deploy the same code in westus2 and southcentralus without issue, but we're seeing this in westcentralus. what region is everyone else deploying to?

a bash curl call to the api (https://docs.microsoft.com/en-us/rest/api/storagerp/queue-services/get-service-properties) works fine in all cases

[fgarcia-cnb]

we figured out our issue. for some reason querying queue properties doesn't go through the azure api, but queries the storage account directly. since we are using private endpoint, we had to open up the nsg to allow port 443 connectivity to the storage account subnet from terraform.

[ronaknnathani]

Any updates on this issue? Experiencing the same error with terraform v0.14.5, azurerm v2.69.0.

[smark88]

Same here I have 443 open and such, no luck

OS="Alpine Linux" (atlantis)
VERSION_ID=3.13.5
TF=0.14.11
azuread=1.6.0
azurerm=2.74.0

The below snippet repeats till 10 minute timeout, exact same as the issue.

2021/09/01 18:44:05 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.test201-eu-north-1.module.vault.azurerm_storage_container.main (orphan)"
2021/09/01 18:44:05 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.test201-eu-north-1.module.vault.azurerm_storage_container.main (orphan)"
2021/09/01 18:44:06 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
...
2021/09/01 18:47:01 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2021/09/01 18:47:02 [TRACE] vertex "module.test201-eu-north-1.module.vault.azurerm_storage_container.main (orphan)": visit complete
2021/09/01 18:47:04 [WARN] Provider "registry.terraform.io/hashicorp/azurerm" produced an unexpected new value for module.test201-eu-north-1.module.vault.azurerm_storage_account.main during refresh.
      - .tags: was null, but now cty.MapValEmpty(cty.String)
2021/09/01 18:47:04 [TRACE] EvalWriteState: using RefreshState for module.test201-eu-north-1.module.vault.azurerm_storage_account.main
2021/09/01 18:47:04 [TRACE] EvalWriteState: writing current state object for module.test201-eu-north-1.module.vault.azurerm_storage_account.main
2021/09/01 18:47:04 [TRACE] EvalWriteDiff: recorded Delete change for module.test201-eu-north-1.module.vault.azurerm_storage_account.main
2021/09/01 18:47:04 [TRACE] EvalWriteState: removing state object for module.test201-eu-north-1.module.vault.azurerm_storage_account.main
2021/09/01 18:47:04 [TRACE] vertex "module.test201-eu-north-1.module.vault.azurerm_storage_account.main (orphan)": visit complete
2021/09/01 18:47:04 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2021/09/01 18:47:04 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" errored, so skipping
2021/09/01 18:47:04 [TRACE] dag/walk: upstream of "root" errored, so skipping
2021/09/01 18:47:04 [INFO] backend/local: plan operation completed
2021-09-01T18:47:04.370Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-09-01T18:47:04.376Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/2.74.0/linux_amd64/terraform-provider-azurerm_v2.74.0_x5 pid=200956
2021-09-01T18:47:04.377Z [DEBUG] plugin: plugin exited

[robertbrandso]

I experienced the same as @fgarcia-cnb mentions, the query didn't go through the public Azure API, but connected to the storage accounts private endpoint. Before I opened tcp-443 in Azure Firewall to the private endpoint I got the error message: Error: shares.Client#GetProperties: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded. After I opened the port everything ran successfully.

I was using the azurerm provider version 2.57.0.

[Tulki]

@robertbrandso or @fgarcia-cnb do you happen to have a terraform snippet for how you got this workaround running? Bit confused as to where that's being set, as I hit the same issue even without a firewall rule at the vnet level, and as far as I know according to the docs, NSG rules are ignored for private endpoints with enforce_private_link_endpoint_network_policies = true set on the subnet.

Thank you!

[fgarcia-cnb]

in our case, its not a tf code issue. its a matter of allowing our VM thats running terraform outbound access to the storage account private endpoint subnet. you are correct, private endpoint doesnt use nsg, but thats on the inbound side.

[robertbrandso]

In the release announcement for azurerm 3.0 it says:

Storage:
We’ll switch to using Resource Manager (management plane) for all resources.

Maybe that'll fix this issue?

[tombuildsstuff]

This'll be fixed by #2977 - please subscribe to that issue for updates

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.