Cannot use remote backend on TFC using terraform 0.14.x+

[sudermanjr]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I am using the pre-release from 12/26, and I still seem to have the issue with remote execution in Terraform versions greater that 1.1 (I have tried several, from 1.1.2, 1.2, and 1.3.6). I expected this to be fixed by #2793

│ Error: Saving a generated plan is currently not supported
│ 
│ The "remote" backend does not support saving the generated execution plan
│ locally at this time.
╵

I was able to work around the issue by overriding the plan workflow with no -out parameter, but the functionality is a bit reduced in Github (the link goes to Atlantis, not terraform cloud)

Reproduction Steps

Using Atlantis image tag v0.22.0-pre.20221226 hooked up to Github, try planning a terraform project that uses terraform cloud as the remote backend, with remote execution enabled. Use terraform 1.3.6 (latest at time of writing)

Logs

I don't have the logs, as they have already been rotated after implementing the workaround

Environment details

Atlantis v0.22.0-pre.20221226 installed via Helm in K8s
Github
Terraform Cloud
TF Version 1.3.6

Additional Context

There have been several issues related to this. Most are linked in #2793

• Terraform Cloud Remote Operations broken for Terraform versions >= 0.15.0 #1628

[nitrocode]

@sudermanjr did this work for you using previous versions like 0.21.0?

cc @lilincmu

[sudermanjr]

I believe it did, yes. The problem popped up after upgrading from 0.21 to a later version.

[nitrocode]

@sudermanjr just to make sure this is related to the change in #2793 which was merged on Dec 15, could you try the pre-release before that change was merged v0.21.1-pre.20221213 ?

If that works as expected, could you also try v0.22.0-pre.20221219, which is the pre-release right after the change, and see if you can reproduce the same issue above?

If you confirm it works for v0.21.1-pre.20221213 and it fails for v0.22.0-pre.20221219, then we'll have to revert #2793 I'd imagine unless @lilincmu (or another contributor) can investigate and resolve the issue.

[sudermanjr]

Interesting. I went back and tried both v0.21.1-pre.20221213 and 0.21.0 and had the same issue with both. So now I'm questioning my own memory.

Edit: This is with terraform 1.3.6 - need to test other tf versions

Edit2: Confirmed not working with terraform 1.1.x on any version tested - 0.21.0, v0.21.1-pre.20221213 , and v0.22.0-pre.20221219

[nitrocode]

Hmm so when was the last atlantis this worked for you?

Or is it a recent problem with the latest terraform versions? Does terraform 1.0.x plan correctly with the remote backend on TFC with the latest Atlantis?

[sudermanjr]

I'm honestly not sure at this point. We have a lot of different terraform repos on a lot of different versions, and I've been making changes quite a lot to try and test this.

1.0.11 doesn't seem to work either:

running "/usr/local/bin/terraform1.0.11 plan -input=false -refresh -out \"/atlantis-data/repos/FairwindsOps/insights-terraform/203/default/global/insights-global-default.tfplan\"" in "/atlantis-data/repos/FairwindsOps/insights-terraform/203/default/global": exit status 1
╷
│ Error: Saving a generated plan is currently not supported
│ 
│ The "remote" backend does not support saving the generated execution plan
│ locally at this time.
╵

[nitrocode]

Has this always failed? If not, can you find a combination of terraform version and atlantis version that does work?

Otherwise, from the looks of it, this would be impossible to diagnose / triage. :(

[sudermanjr]

Terraform 0.13.7 works with the latest pre-release

[nitrocode]

So perhaps this is a terraform issue and not an atlantis issue then?

[sudermanjr]

The terraform repos work just fine when I don't use atlantis and run from the CLI or use Terraform's VCS integration. I can get Atlantis to work with custom workflows, but the links in Github don't get updated with the remote URL.

Honestly, at this point we are most likely going to remove Atlantis entirely and just stick with terraform cloud. They have updated the VCS integration enough to make the slightly degraded git experience acceptable, and I have not had a consistently working Atlantis configuration across all repos in months.

Appreciate you trying to track this down and all your time.

[nitrocode]

I understand your frustration. You have created this issue and it's a real issue even if it no longer affects you. Let's keep it open and re-title it so everyone knows that this is a problem for terraform 0.14.x and higher.

cc: @lilincmu in case you have run into this issue or you have thoughts on this.

[nitrocode]

@sudermanjr actually, this is also worth reviewing. This other person had the same issue and it turned out to be a setting in his pipeline that needed to be enabled.

https://discuss.hashicorp.com/t/saving-a-generated-plan-is-currently-not-supported/2116

[sudermanjr]

Thanks for the link, but we've been using remote execution from the beginning for various security reasons. My understanding is that Atlantis is supposed to detect this error and modify how the workflow behaves in order to compensate. This used to work quite well, where we used the default workflow with remote execution and Atlantis would update the link in the github PR to point to TFCloud and then also share the plan in the PR.

[nitrocode]

Ah interesting. Also did you know you can get more verbose logs by running the following?

atlantis plan --verbose

That would also help for someone in the future to contribute a PR to resolve this issue. 🙏

[sudermanjr]

Not much there to be honest, but here's a redacted version:

[DBUG] 3 files were modified in this pull request
[DBUG] got workspace lock
[INFO] Refreshing git tokens for Github App
[INFO] updated git app credentials in /home/atlantis/.git-credentials
[INFO] successfully ran git config --REDACTED credential.helper store
[INFO] successfully ran git config --REDACTED url.https://[email protected] ssh://[email protected]
[DBUG] clone directory "/atlantis-data/repos/REDACTED/REDACTED/203/default" already exists, checking if it's at the right commit
[DBUG] repo is at correct commit "3ea28896a7900c60586808c49a6c3921243dd2e3" so will not re-clone
[INFO] successfully parsed atlantis.yaml file
[DBUG] checking if project at dir "REDACTED" workspace "default" was modified
[DBUG] file "REDACTED/versions.tf" matched pattern
[DBUG] checking if project at dir "REDACTED" workspace "default" was modified
[DBUG] checking if project at dir "REDACTED" workspace "default" was modified
[DBUG] checking if project at dir "REDACTED" workspace "default" was modified
[INFO] 1 projects are to be planned based on their when_modified config
[DBUG] determining config for project at dir: "REDACTED" workspace: "default"
[DBUG] MergeProjectCfg started
[DBUG] setting delete_source_branch_on_merge: false from default server config
[DBUG] setting repo_locking: true from default server config
[DBUG] setting apply_requirements: [approved,mergeable] from repos[1], id: /.*/
[DBUG] setting workflow: "default" from default server config
[DBUG] setting allowed_overrides: [workflow] from repos[1], id: /.*/
[DBUG] setting allow_custom_workflows: true from repos[1], id: /.*/
[DBUG] MergeProjectCfg completed
[DBUG] final settings: apply_requirements: [approved,mergeable], workflow: default
[DBUG] Building project command context for plan
[DBUG] deleting previous plans and locks
[INFO] acquired lock with id "REDACTED/REDACTED/REDACTED/default"
[DBUG] acquired lock for project
[INFO] Refreshing git tokens for Github App
[DBUG] git credentials file has expected contents, not modifying
[DBUG] clone directory "/atlantis-data/repos/REDACTED/REDACTED/203/default" already exists, checking if it's at the right commit
[DBUG] repo is at correct commit "3ea28896a7900c60586808c49a6c3921243dd2e3" so will not re-clone
[DBUG] starting "/usr/local/bin/terraform1.0.11 init -input=false" in "/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED"
[INFO] successfully ran "/usr/local/bin/terraform1.0.11 init -input=false" in "/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED"
[INFO] successfully ran "/usr/local/bin/terraform1.0.11 workspace show" in "/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED"
[DBUG] starting "/usr/local/bin/terraform1.0.11 plan -input=false -refresh -out \"/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED/insights-REDACTED-default.tfplan\"" in "/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED"
[EROR] running "/usr/local/bin/terraform1.0.11 plan -input=false -refresh -out \"/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED/insights-REDACTED-default.tfplan\"" in "/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED": exit status 1
[EROR] Error running plan operation: running "/usr/local/bin/terraform1.0.11 plan -input=false -refresh -out \"/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED/insights-REDACTED-default.tfplan\"" in "/atlantis-data/repos/REDACTED/REDACTED/203/default/REDACTED": exit status 1
╷
│ Error: Saving a generated plan is currently not supported
│ 
│ The "remote" backend does not support saving the generated execution plan
│ locally at this time.
╵

[INFO] deleting plans because there were errors and automerge requires all plans succeed

[nitrocode]

Thank you. Looks like this is a duplicate of this issue.

There is a workaround here for now.

#2794 (comment)