terraform destroy does not work in non-trivial cases

[bitglue]

As I develop increasingly complicated Terraform configurations, I find that destroy pretty much never works. I haven't nailed down a trivial repro case yet, but since I don't think that will be easy I wanted to see if this was a known issue or if anyone had some insight before I put more time into it.

The issue seems to be something with the graph walking order when there are modules. See this plan:

Or its transitive reduction, which is somewhat easier to read:

In particular, fleet_unit.login and fleet_unit.ssl_terminator should be destroyed before aws_instance.coreos, which is important because fleet is a service provided by those instances. That's not what happens, though: first Terraform deletes the AWS instances, then when it tries to delete the fleet units the provider errors because the API it's using no longer exists.

An odd thing I note: there's actually only one fleet provider and one aws provider, and it's in the root module. I'm not sure why the graph contains duplicate providers inside the modules, and only sometimes, but maybe that's a hint to what's going on.

After destroy completes with those errors, the state becomes all fowled up. refresh will complain:

Error refreshing state: Resource 'aws_security_group.internet' not found for variable 'aws_security_group.internet.id'

At this point I can't get Terraform to do much of anything, and I just have to clean up the remaining resources with the AWS console, nuke terraform.tfstate, and start fresh.

I realise this is a pretty crappy bug report. I'll do my best to reduce it further or fix it -- I just wanted some feedback first. If there are any hints at what code might be responsible that would save me a lot of time.

[mitchellh]

Thanks! I would be very interested to see what happens when we merge in the f-ast-graph branch that I'm working on regarding this. I'm sure the issue will likely still exist, but we'll get much better insight into what is happening (we'll be recording each graph transformation for debugging) and it'll be easier to fix that way.

[bitglue]

Cool. I don't know any way on GitHub to watch for a branch to land, but if you give me a nudge when it does I'll do some more testing and see if I can isolate this failure better.

[julienba]

Hi,

I've created a minimal sample project that highlight the issue, if that can help you
https://github.com/julienba/terraform-module-sample

[bitglue]

So as it happened I had reason to look into this more, and it looks like currently terraform graph -module-depth=1 doesn't actually expand the graph anymore. The graph has nodes named things like "module.public_dev_coreos (expanded)", but actually nothing is expanded.

Now that it looks like f-ast-graph has landed, is there something more I can do to determine what's going on behind the scenes?

[bitglue]

I haven't figured out all the code yet, but I suspect what's going on is that a destroy walks the modules in the correct order, but then walks the components of each module in the opposite order, that is, the order that should be used to create the module. Of course, all the edges need to be flipped in the destroy case, and when they aren't, it pretty much guarantees that destroy will fail since dependencies are deleted before their dependents, for example subnets are deleted before the instances running in them.

This is based on my observation of the output of terraform destroy: the thing it always tries to delete first, and which always fails, is the most foundational unit of the module, or the thing that should be created first.

[mitchellh]

The overarching title of this bug may not be fixed, but at least the specific case mentioned in this issue has been fixed with #1781. As @bitglue expected, modules were being treated as a single unit and this is no longer the case in the graph. This should fix this issue.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.