Regression: terraform init can no longer migrate state local to remote, unless versions precisely match - Remote workspace Terraform version "X" does not match local Terraform version "Y"

[maxb]

Use of terraform init to move state between backends is an advanced feature, but vital in some maintenance scenarios. Unfortunately this has regressed in Terraform 0.15 and later.

Using terraform init to migrate state from a local backend to a remote one mostly no longer works, because it uses the fallback version checking at

terraform/internal/backend/remote/backend.go

Lines 638 to 650 in 0fcb750

	// This is a fallback error check. Most code paths should use other
	// mechanisms to check the version, then set the ignoreVersionConflict
	// field to true. This check is only in place to ensure that we don't
	// accidentally upgrade state with a new code path, and the version check
	// logic is coarser and simpler.
	if !b.ignoreVersionConflict {
	wsv := workspace.TerraformVersion
	// Explicitly ignore the pseudo-version "latest" here, as it will cause
	// plan and apply to always fail.
	if wsv != tfversion.String() && wsv != "latest" {
	return nil, fmt.Errorf("Remote workspace Terraform version %q does not match local Terraform version %q", workspace.TerraformVersion, tfversion.String())
	}
	}

which is too simplistic for real world uses.

Terraform Version

Found in Terraform 1.0.9, but I think all versions containing the code from #26947, i.e. 0.15 and later.

Steps to Reproduce

To reproduce, you need:

• Any Terraform configuration at all - a single empty null_resource would do
• Set up in a TFE/TFC workspace so that you can use the remote backend
• A terraform { backend "remote" { ... } } block in your configuration so terraform init knows which TFE/TFC to talk to
• To be logged in (terraform login) to the TFE/TFC
• To be using a local Terraform CLI version which is not an exact match for the version configured in the TFE/TFC workspace. (For example, I used loacl Terraform CLI version 1.0.9 whilst my TFC workspace was set to 1.0.8)

1. terraform init (this sets up the remote backend defined in configuration)
2. Delete the terraform { backend "remote" { ... } } block from your configuration - the default local backend now applies
3. terraform init -migrate-state -force-copy to fetch the state from the remote backend and save it locally
4. In a real world situation, you would now run various commands to change the state prior to returning it to the remote backend - for this bug reproduction, we don't need to actually do any state changes, carry on to the next step
5. Revert the configuration change from step 2. - re-add the terraform { backend "remote" { ... } } pointing at the remote backend
6. terraform init -migrate-state -force-copy

Expected Behavior

Step 6 above migrates local state to the remote backend

Actual Behavior

Step 6 fails:

Initializing the backend...
╷
│ Error: Error loading state:
│     Remote workspace Terraform version "1.0.8" does not match local Terraform version "1.0.9"
│ 
│ Terraform failed to load the default state from the "remote" backend.
│ State migration cannot occur unless the state can be loaded. Backend
│ modification and state migration has been aborted. The state in both the
│ source and the destination remain unmodified. Please resolve the
│ above error and try again.
│ 
│ 
╵

Additional Context - Problem is worse in TFE environment

In the above report, I have reproduced the problem using TFC. With TFC, the relatively simple workaround of "Just use the exactly matching Terraform CLI version that the workspace is configured to use" exists.

However, I actually discovered this at work, with our TFE environment, where the issue is much more blocking: in our TFE environment, workspace Terraform version strings don't exactly match any Terraform CLI version, because they identify a bundle containing the Terraform CLI and providers - and so include extra detail identifying the providers they contain. Therefore, the workaround of using a Terraform CLI version that exactly matches the workspace version is impossible, as no such version exists.

Additional Bug - -ignore-remote-version option does not work to bypass this error

The -ignore-remote-version option is supposed to bypass checks like this. However, whilst it is applied to a remote backend during normal operation, the Terraform CLI fails to apply the option value to a remote backend being used in a state migration operation.

[alisdair]

Thanks for reporting this! I'm able to confirm the behaviour you're seeing, although it's not immediately obvious to me how this is happening.

[alisdair]

The fix in #29793 has been merged into the v1.0 branch, and will go out with Terraform 1.0.10.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.