Error running aztfexport in RHEL8

[cameronew]

Hi,

Having the following issue when using the tool. Terraform init works fine but it seems to have trouble when running a simple export.
See error below. The plug does seem to be installed ok.

aztfexport resource-group mytestrg
#error message
Microsoft Azure Export for Terraform

initializing provider for the import directories: 2 errors occurred:
* task error: error running terraform init: exit status 1

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.65.0: unexpected EOF

    * task error: error running terraform init: exit status 1

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.65.0: unexpected EOF

#plugin
-rwxr-xr-x 1 ewan ewan 194940928 Oct 5 12:55 terraform-provider-azurerm_v3.65.0_x5
[ewan@00f836ff2ebd linux_amd64]$ pwd
/home/ewan/test/.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.65.0/linux_amd64

#terraform init seems ok
[ewan@00f836ff2ebd test]$ terraform init

Initializing the backend...

Initializing provider plugins...

• Reusing previous version of hashicorp/azurerm from the dependency lock file
• Using previously-installed hashicorp/azurerm v3.65.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[ewan@00f836ff2ebd test]$

[magodo]

@cameronew This seems to be a network issue when downloading the provider. Did you try to remove the pre-installed provider in your environment, and download the provider again via terraform init?

[cameronew]

@cameronew This seems to be a network issue when downloading the provider. Did you try to remove the pre-installed provider in your environment, and download the provider again via terraform init?

@magodo Yes I deleted the provider and ran terraform init which worked fine then had th
log.txt
e same issue with azfexport. Attached is the trace log.

[magodo]

Do you have any special settings in your .terraformrc?

[cameronew]

No just the defaults

[magodo]

@cameronew Just want to check if you still have this issue, or I can close it now?

[ryanhitch]

I am seeing the same issue on RHEL9. It looks like /tmp is filling up. My /tmp is 2GB, however aztfexport seems to be repeatedly downloading the same provider into /tmp multiple times which fills it up. Running terraform init in the same directory works fine, so I am not sure what issue aztfexport is having. Any suggestions?

Microsoft Azure Export for Terraform

initializing provider for the import directories: 5 errors occurred:
* task error: error running terraform init: exit status 1

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.99.0: write
.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.99.0/linux_amd64/terraform-provider-azurerm_v3.99.0_x5:
no space left on device

    * task error: error running terraform init: exit status 1

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.99.0: write
.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.99.0/linux_amd64/terraform-provider-azurerm_v3.99.0_x5:
no space left on device

    * task error: error running terraform init: exit status 1

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.99.0: write
.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.99.0/linux_amd64/terraform-provider-azurerm_v3.99.0_x5:
no space left on device

    * task error: error running terraform init: exit status 1

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.99.0: write
.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.99.0/linux_amd64/terraform-provider-azurerm_v3.99.0_x5:
no space left on device

    * task error: error running terraform init: exit status 1

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.99.0: write
.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.99.0/linux_amd64/terraform-provider-azurerm_v3.99.0_x5:
no space left on device

[ryanhitch]

I am not sure if this is related, but I tried the same thing on a Windows machine and get a step further, however now it appears that aztfexport is only configured to interact with the Azure Commercial Cloud API (management.azure.com). The subscription that I want to access is the Azure US Government Cloud, which uses this API URL management.usgovcloudapi.net. az cli uses the following to set this, and the terraform azurerm provider also supports this with the metadata_host parameter. Is there a way to set it with aztfexport?

az cloud set --name "AzureUSGovernment"

[magodo]

@ryanhitch The first error is due to:

no space left on device

The second ask is now supported by: #546

[ryanhitch]

@magodo Before I run aztfexport 1.9G is available in /tmp. "no space left on device" is caused by aztfexport downloading the azurerm provider repeatedly until there is no space left in /tmp. terraform init in the same directory just downloads the provider once, so I am not sure what aztfexport is doing.

Thanks for #546

[magodo]

@ryanhitch Instead you can configure the terraform to cache the provider to avoid repeated download.

[ryanhitch]

By default it looks like aztfexport downloads 10 copies of the azurerm provider into /tmp when running 'aztfexport rg myResourceGroup' which I am guessing is to export 10 resources at a time.
This uses 2.3 GB since the provider is ~228MB right now.
It looks like the solution is just to ensure that /tmp has at least 2.3 GB free or to cache the provider as @magodo suggests above.

A default Azure RHEL9 VM image only has a 1GB /tmp partition which is likely why some people have had this issue.

2024-08-19T17:37:58.711Z [INFO] aztfexport: New resource group meta
2024-08-19T17:37:58.712Z [INFO] aztfexport: Init Terraform
2024-08-19T17:37:58.773Z [INFO] aztfexport: Find terraform binary at /usr/bin/terraform
2024-08-19T17:37:58.773Z [INFO] aztfexport: Init provider
2024-08-19T17:37:58.773Z [INFO] aztfexport: Output directory doesn't contain provider setting, create one then
2024-08-19T17:37:58.773Z [INFO] aztfexport: Output directory doesn't contain terraform block, create one then
2024-08-19T17:37:58.773Z [DEBUG] aztfexport: Run "terraform init" for the output directory /home/user/outdir
2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-197078599
2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-643364438
2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-563213887
2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-242333829
2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-3144448852
2024-08-19T17:38:01.115Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-4214556662
2024-08-19T17:38:01.116Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-1222360874
2024-08-19T17:38:01.116Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-3816848619
2024-08-19T17:38:01.116Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-1915249044
2024-08-19T17:38:01.116Z [DEBUG] aztfexport: Run "terraform init" for the import directory /tmp/aztfexport-2642539073
2024-08-19T17:38:11.071Z [ERROR] aztfexport: aztfexport ends with error: initializing provider for the import directories: 10 errors occurred:
* task error: error running terraform init: exit status 1

Error: Failed to install provider

Error while installing hashicorp/azurerm v3.99.0: write
/tmp/terraform-provider101873945: no space left on device

[magodo]

Another way is to reduce the parallelism to a lower number (defaults to 10).