Atlantis fails to create new stack workspaces with s3 backend

[xarses]

After updating to runatlantis/atlantis:latest ~ 0.17-beta we found that we where unable to create new workspaces beyond default. Creation of workspaces would fail with

running "/app/.atlantis/bin/terraform0.14.7 init -input=false -no-color -upgrade" in "/app/.atlantis/repos/org/repo/1234/dev/groups/stack": exit status 1
Upgrading modules...
...

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

The currently selected workspace (dev) does not exist.
  This is expected behavior when the selected workspace did not have an
  existing non-empty state. Please enter a number to select a workspace:
  
  1. default

  Enter a value: 

Error: Failed to select workspace: input not a valid number

This reproduces with terraform 0.14.7 0.14.10 and 0.13.6

It seems that introducing TF_WORKSPACE in #1363 causes a long standing bug with terraform to resurface (hashicorp/terraform#26127)

This can be worked around by causing TF_WORKSPACE to become unset. This is accomplished by changing the workflow env

workflows:
  default:
    plan:
      steps:
      - env:
          name: TF_WORKSPACE
          value:

[xbglowx]

Seeing this as well with atlantis v0.16.1 and terraform 0.13.5.

[davidmontoyago]

Can confirm same issue with atlantis v0.16.1, terraform 0.13.6 and GCS backend

[monitrue]

@nishkrishnan @lkysow Do you mind if I raise a PR to revert #1363? I believe it did more harm than good by setting a custom Terraform control variable to a non-default value and must be reverted.

It broke Terraform workspaces which now require a workaround to work at all.

Users requiring TF_WORKSPACE to be set are free to set it manually via a custom workflow or system-wide and have full control over the value. I see no reason why it should be set by Atlantis to a fixed non-default value, in the same manner as other TF_ variables are neither set nor changed by Atlantis.

Making this change might break some existing workflows that rely on #1363, in the same fashion as #1363 broke my and other people's workflows. I believe if it is clearly mentioned in Release Notes as a breaking change (unlike #1363 which was a surprise), the impact should be minimal.

[grimm26]

The workaround does not work for me. I set TF_WORKSPACE to an empty value in both plan and apply in my workflow. Plan works, apply throws this if I try to set my own workspace in atlantis.yaml:

building command for dir ".": running commands in workspace "pr223" is not allowed because this directory is only configured for the following workspaces: default

If I don't set a workspace:

building command for dir "blah/blah": yaml: unmarshal errors:
  line 6: field workspace already set in type raw.Project

[monitrue]

building command for dir ".": running commands in workspace "pr223" is not allowed because this directory is only configured for the following workspaces: default

Apparently your atlantis apply is executed in the wrong default workspace in the wrong directory. Probably your atlantis plan custom workflow does not init the Terraform workspace or your atlantis.yaml does not contain the correct workspace. It does not have to do with the workaround, it is that making Terraform workspaces to work with Atlantis is a bit tricky in general.

Atlantis starts its atlantis plan workflow in the default directory in the default workspace. On plan - init, if Atlantis detects a workspace request (via -w or in atlantis.yaml), it creates a new directory for the workspace, checks out the PR there, initializes the workspace and runs the rest of the plan in this new directory. On atlantis apply it only runs in the correct workspace directory.

It is criticial that your custom workflow runs - init as the first step, otherwise the workspace and new directory for it is not created (what happens in your case) and things do not work as expected. A simple example that works for me:

Repo-level atlantis.yaml:

version: 3
automerge: false
projects:
- name: dev
  dir: .
  workflow: dev
  workspace: dev
  autoplan:
    enabled: true

Custom workflow in repos.yaml:

workflows:
  dev:
    plan:
      steps:
      - init
      - plan
    apply:
      steps:
      - apply

[marcelobartsch-jt]

The problem seems to happens before the apply workflow is called, in fact the error seems to came from buildProjectApplyCommand in project_command_builder.go as it expect the workspae to be passed on the CommentCommand, but if is not passed it assumes the DefaultWorkspace (which is 'default') instead of reading that name from the actual workspace in use by terraform.