How do I manually destroy resources in AWS if terraform destroy fails?

[tylerpotts]

Document how to destroy resources after looking them up by tag in AWS

[tylerpotts]

Somewhat related to #579

[tylerpotts]

AWS will prevent some resources from being destroyed if there are dependent resources that still exist. The order of resource deletion is below:

Basic steps:

• delete EFS
• delete EKS nodegroups
• delete EKS cluster
• delete security group

[rsignell-usgs]

Yes, these basic steps worked for me!

[brl0]

@iameskild, You've been through this a couple of times recently, it would be good to add anything else that you think should be checked to this issue.

[tylerpotts]

Converting to a discussion for future posterity

[rsignell-usgs]

@iameskild or other qhubbers:

I want to destroy all the resources for our qhub=0.13.4 deployment on AWS.
(we've deployed qhub=0.14.1 on another AWS account, so we are moving on!)

I was planning destroy the old one by:

• cloning our private github repo with the deployment/config
• typing "cd qhub/infrastructure; terraform destroy" ?

So two questions:

1. Is that the right approach?
2. Do I need to worry about having the right version of "terraform" or "qhub" installed in my conda environment?

(I didn't see anything about destroying qhub in the docs)

[iameskild]

Hey @rsignell-usgs, @viniciusdc is correct. I would just like to add that this would also be a good time for a backup (should you feel you need one).

[rsignell-usgs]

We have a completely separate repo and cluster name (esip-spruce => esip-maple) for the new one. So the destroy should be okay?
Any special version of terraform required for 0.3.14?

[iameskild]

If you qhub==0.3.14 installed locally, qhub should handle the terraform install and version. I believe it was 1.0+ at that point.

[rsignell-usgs]

Okay! And I just realized there is no way the terraform destroy command can destroy the resources on the new cluster because it's on a different AWS account, right?

[iameskild]

So long as you're new AWS account credentials do not take precedent over your older ones(see this ordering), I think you should be fine.

If you run qhub destroy -c qhub-config.yaml, the most important thing to check might be that you have the correct qhub-config.yaml selected.

[rsignell-usgs]

For some reason I got permission denied on the scripts, even though I had my AWS credentials specified via env vars. So I used the console route again.

A couple of additional notes:

• deleting the node groups is kind of tricky, as you need to go to EKS, then select a cluster, then configuration, then compute and then select each of the 3 node groups for each cluster one by one. And then I had to wait a good 10 minutes for that to complete before I could delete the cluster.
• to delete the security group, you need to go to IAM first, then to security groups.
• I went to S3 and deleted the qhub-terraform-state bucket