Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Terraform v1.4 #124

Merged
merged 3 commits into from
Mar 9, 2023
Merged

Add support for Terraform v1.4 #124

merged 3 commits into from
Mar 9, 2023

Conversation

minamijoyo
Copy link
Owner

@minamijoyo minamijoyo commented Feb 13, 2023

There are no changes in implementation, but some adjustments have been made to the test settings:

(1) Add terraform 1.4.0 to the test matrix
(2) Use a local filesystem mirror on CI
Starting with Terraform v1.4, the global plugin cache is ignored on the first terraform init. This makes caching in CI meaningless: hashicorp/terraform#32129, hashicorp/terraform#32494

To utilize the cache, we use a local filesystem mirror. Strictly speaking, the mirror is only available in Terraform v0.13+, but it is hard to compare versions in bash, so we use the mirror unless v0.x.
https://developer.hashicorp.com/terraform/cli/config/config-file#implied-local-mirror-directories

(3) Export TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE=true in CI
Starting from Terraform v1.4, launching terraform providers in the acceptance test has been failing more frequently with a text file busy error.

--- FAIL: TestAccMultiStateMigratorApplySimple (1.07s)
    multi_state_migrator_test.go:123: failed to run terraform init: failed to run command (exited 1): terraform init -input=false -no-color
        stdout:

        Initializing the backend...

        Successfully configured the backend "s3"! Terraform will automatically
        use this backend unless the backend configuration changes.

        Initializing provider plugins...
        - Finding latest version of hashicorp/null...
        - Installing hashicorp/null v3.2.1...

        stderr:

        Error: Failed to install provider

        Error while installing hashicorp/null v3.2.1: open
        /tmp/plugin-cache/registry.terraform.io/hashicorp/null/3.2.1/linux_amd64/terraform-provider-null_v3.2.1_x5:
        text file busy

After some investigation, I found Go's os/exec.Cmd.Run() does not wait for the grandchild process to complete; from the point of view of tfmigrate, the terraform command is the child process, and the provider is the grandchild process.
golang/go#23019

If I understand correctly, this is not a Terraform issue and theoretically should occur in versions older than v1.4; the changes in v1.4 may have broken the balance of execution timing and made the test very flaky. I experimented with inserting some sleep but could not get the test to stabilize correctly. After trying various things, I found that the test became stable by enabling the TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE flag was introduced in v1.4. This is an escape hatch to revert to the v1.3 equivalent of the global cache behavior change in v1.4.
hashicorp/terraform#32726

This behavior change has already been addressed in the previous commit using a local file system mirror, so activating this flag does not seem to make any sense. Even though I have no other reasonable solutions now, please let me know if anyone finds a better solution.

Starting with Terraform v1.4, the global plugin cache is ignored on the
first terraform init. This makes caching in CI meaningless.
hashicorp/terraform#32494

To utilize the cache, we use a local filesystem mirror. Strictly
speaking, the mirror is only available in Terraform v0.13+, but it is
hard to compare versions in bash, so we use the mirror unless v0.x.
https://developer.hashicorp.com/terraform/cli/config/config-file#implied-local-mirror-directories
Starting from Terraform v1.4, launching terraform providers in the
acceptance test has been failing more frequently with a text file busy
error.

```
--- FAIL: TestAccMultiStateMigratorApplySimple (1.07s)
    multi_state_migrator_test.go:123: failed to run terraform init: failed to run command (exited 1): terraform init -input=false -no-color
        stdout:

        Initializing the backend...

        Successfully configured the backend "s3"! Terraform will automatically
        use this backend unless the backend configuration changes.

        Initializing provider plugins...
        - Finding latest version of hashicorp/null...
        - Installing hashicorp/null v3.2.1...

        stderr:

        Error: Failed to install provider

        Error while installing hashicorp/null v3.2.1: open
        /tmp/plugin-cache/registry.terraform.io/hashicorp/null/3.2.1/linux_amd64/terraform-provider-null_v3.2.1_x5:
        text file busy
```

After some investigation, I found Go's `os/exec.Cmd.Run()` does not wait
for the grandchild process to complete; from the point of view of
tfmigrate, the terraform command is the child process, and the provider
is the grandchild process.

golang/go#23019

If I understand correctly, this is not a Terraform issue and
theoretically should occur in versions older than v1.4; the changes in
v1.4 may have broken the balance of execution timing and made the test
very flaky. I experimented with inserting some sleep but could not get
the test to stabilize correctly. After trying various things, I found
that the test became stable by enabling the
`TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE` flag was introduced in
v1.4. This is an escape hatch to revert to the v1.3 equivalent of the
global cache behavior change in v1.4.
hashicorp/terraform#32726

This behavior change has already been addressed in the previous commit
using a local file system mirror, so activating this flag does not seem
to make any sense. Even though I have no other reasonable solutions now,
please let me know if anyone finds a better solution.
@minamijoyo minamijoyo changed the title [WIP] Add support for Terraform v1.4 Add support for Terraform v1.4 Mar 9, 2023
@minamijoyo minamijoyo merged commit bd115e2 into master Mar 9, 2023
@minamijoyo minamijoyo deleted the tf-1.4 branch March 9, 2023 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant