Cost Management Export for custom timeframe fails with 401 Unauthorized and points to trafficmanager.net endpoint

[BeeDeeBePope]

I'm deploying a solution that uses Subscription scoped Cost Management Exports using Azure DevOps.
I'm using azapi 1.14.0 to create the exports with a Custom timeframe(azurerm does not support it at the moment). When running locally terraform apply works fine, but in automation, it fails with the following error for each subscription:

##[error]Terraform command 'apply' failed with exit code '1'.
##[error]╷
│ Error: Failed to create/update resource
│ 
│   with azapi_resource.cost_management_export["4"],
│   on cost-management-exports.tf line 38, in resource "azapi_resource" "cost_management_export":
│   38: resource "azapi_resource" "cost_management_export" {
│ 
│ creating/updating Resource: (ResourceId
│ "/subscriptions/<radacted-subscription-id>/providers/Microsoft.CostManagement/exports/usageExport"
│ / Api Version "2022-10-01"): GET
│ https://costmanagement.trafficmanager.net/subscriptions/<radacted-subscription-id>/providers/Microsoft.CostManagement/exports/usageExport
│ --------------------------------------------------------------------------------
│ RESPONSE 401: 401 Unauthorized
│ ERROR CODE: 401
│ --------------------------------------------------------------------------------
│ {
│   "error": {
│     "code": "401",
│     "message": "Request is missing a certificate or valid token"
│   }
│ }
│ --------------------------------------------------------------------------------
│ 
╵

The azapi_resource definition:

resource "azapi_resource" "cost_management_export" {
  for_each  = { for index, value in var.usage_export.usage_export_subscriptions : index => value }
  type      = "Microsoft.CostManagement/exports@2022-10-01"
  name      = var.usage_export.usage_export_name
  parent_id = each.value

  body = {
    properties = {
      schedule = {
        status     = "Inactive"
      }
      format = "Csv"
      deliveryInfo = {
        destination = {
          resourceId     = azurerm_storage_account.usage_exports.id
          container      = azurerm_storage_container.subscription_exports.name
          rootFolderPath = each.key
        }
      },
      definition = {
        type      = "ActualCost"
        timeframe = "Custom"
        timePeriod = {
          from = "2024-08-01T00:00:00Z"
          to   = "2024-08-09T00:00:00Z"
        }
        dataSet = {
          granularity = "Daily"
        }
      }
    }
  }
}

Provider definitions:

provider "azurerm" {
  subscription_id = var.hub.subscription_id
  features {}
}

provider "azapi" {
}

terraform {
  required_providers {
    azapi = {
      source = "azure/azapi"
    }
  }
  backend "azurerm" {
    key = "usage-export.tfstate"
  }
}

Other resources successfully provision with azurerm so I assume the service connection is configured correctly. Also tried creating scheduled subscription exports with azurerm, so the Service Principal should have the correct access permissions.

Am I missing some configuration?
Why is it trying to access the trafficmanager.net endpoint?
https://costmanagement.trafficmanager.net/subscriptions//providers/Microsoft.CostManagement/exports/usageExport

[ms-henglu]

Hi @BeeDeeBePope ,

Thank you for taking time to report this issue!

This issue is caused by the upstream API, and was fixed in #184 (comment).

Please double confirm that in the automation whether the azapi provider version is above 1.9.0.

[BeeDeeBePope]

Hi @ms-henglu,
I'm uploading terraform.lock.hcl file to git so the provider versions in automation should be the same as locally.
Anyway, I'm using terraform v1.9.3, and here is the output from terraform init:

Initializing provider plugins...
- Reusing previous version of hashicorp/time from the dependency lock file
- Reusing previous version of hashicorp/azurerm from the dependency lock file
- Reusing previous version of azure/azapi from the dependency lock file
- Installing hashicorp/time v0.12.0...
- Installed hashicorp/time v0.12.0 (signed by HashiCorp)
- Installing hashicorp/azurerm v3.101.0...
- Installed hashicorp/azurerm v3.101.0 (signed by HashiCorp)
- Installing azure/azapi v1.14.0...
- Installed azure/azapi v1.14.0 (signed by a HashiCorp partner, key ID 6F0B91BDE98478CF)

[ms-henglu]

Thanks @BeeDeeBePope - I couldn't reproduce this issue locally and I tried authenticate with Azure by azure cli and service principal.

Here's the config:

resource "azapi_resource" "resourceGroup" {
  type     = "Microsoft.Resources/resourceGroups@2021-04-01"
  name     = "henglu815"
  location = "westus"
  body     = {} # Resource group does not require additional properties in the body
}

resource "azurerm_storage_account" "test" {
  name                     = "henglu815"
  resource_group_name      = azapi_resource.resourceGroup.name
  location                 = azapi_resource.resourceGroup.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "test" {
  name                 = "henglu815"
  storage_account_name = azurerm_storage_account.test.name
}

resource "azapi_resource" "cost_management_export" {
  type      = "Microsoft.CostManagement/exports@2022-10-01"
  name      = "henglu815"
  parent_id = azapi_resource.resourceGroup.id

  body = {
    properties = {
      schedule = {
        status = "Inactive"
      }
      format = "Csv"
      deliveryInfo = {
        destination = {
          resourceId     = azurerm_storage_account.test.id
          container      = azurerm_storage_container.test.name
          rootFolderPath = "test"
        }
      },
      definition = {
        type      = "Usage"
        timeframe = "Custom"
        timePeriod = {
          from = "2024-08-01T00:00:00Z"
          to   = "2024-08-09T00:00:00Z"
        }
        dataSet = {
          granularity = "Daily"
        }
      }
    }
  }
}

[BeeDeeBePope]

Indeed it works fine when run locally. The error occurs when I run a pipeline in Azure DevOps to deploy these resources.

[ms-henglu]

Would you please get the terraform logs from the pipeline?

Here's how:

1. Add the below environment variables.

TF_LOG=DEBUG
TF_LOG_PATH=./terraform.log

2. The logs will stored in the terraform.log file.

[BeeDeeBePope]

Hi @ms-henglu,
I was able to get the debug log from the pipeline. The log file contains some sensitive data and is pretty long for editing the secrets by hand. Do you need the full file? Can you suggest any tool to sanitize it?

[BeeDeeBePope]

@ms-henglu
I modified the terraform configuration to only the necessary resources and changed the log file not to include what I believe are secrets. What is the preferred way to share the file?

[ms-henglu]

Hi @BeeDeeBePope , thanks! You could share it by github gist, or if you want to make it more private, you could upload it to a private github repo and add me as a reader.

[BeeDeeBePope]

Hi @ms-henglu, sorry for the delay. I have created the repository and sent an invitation

[ms-henglu]

Thanks @BeeDeeBePope for the logs! I've opened this PR to fix it: #616

[BeeDeeBePope]

Hi @ms-henglu. Thank you for your work! How can I verify if that works on my end? Is there already a release I can check?