unknown command "node-pools" for "kube-aws" with v0.9.5-rc.6

[cmcconnell1]

Hello
I'm running into an issue wherein I'm not able to delete the node pool according to the docs tagged with this release version as noted below.

Please let me know if I'm missing something.
Thanks!

kube-aws version
kube-aws version v0.9.5-rc.6

According to the docs tagged for this release Destroy the cluster

If you created any node pool, you must delete these first by running kube-aws node-pools destroy --node-pool-name or kube-aws destroy will end up failing because node pools still references AWS resources managed by the main cluster.

so going from the docs, the command to delete the node pool should be:

kube-aws node-pools destroy --node-pool-name foo

However, when I try to run that command, it seems like it's not a valid sub-command/option?

kube-aws node-pools destroy --node-pool-name nodepool1
Error: unknown command "node-pools" for "kube-aws"
Run 'kube-aws --help' for usage.

kube-aws --help
Manage Kubernetes clusters on AWS

Usage:
  kube-aws [command]

Available Commands:
  calculator  Discovery the monthly cost of your cluster
  destroy     Destroy an existing Kubernetes cluster
  init        Initialize default node pool configuration
  render      Render deployment artifacts
  status      Describe an existing Kubernetes cluster
  up          Create a new Kubernetes cluster
  update      Update an existing Kubernetes cluster
  validate    Validate cluster assets
  version     Print version information and exit

Use "kube-aws [command] --help" for more information about a command.

I know this worked before testing node-pools with a previous version and validating from my shell history

history | grep 'node-pools destroy' | awk '{print $2,$3,$4,$5,$6}' | uniq
kube-aws node-pools destroy --node-pool-name first-pool-in-1a
kube-aws node-pools destroy --node-pool-name second-pool-in-1b
kube-aws node-pools destroy --node-pool-name aergo-kube-nodepool-1a
kube-aws node-pools destroy --node-pool-name aergo-kube-nodepool-1b
kube-aws node-pools destroy --node-pool-name nodepool1

I still had an older version of kube-aws to validate this with

/usr/local/bin/kube-aws-v0.9.3-rc.5 --help
Manage Kubernetes clusters on AWS

Usage:
  kube-aws [command]

Available Commands:
  destroy     Destroy an existing Kubernetes cluster
  init        Initialize default node pool configuration
  node-pools  Manage node pools
  render      Render deployment artifacts
  status      Describe an existing Kubernetes cluster
  up          Create a new Kubernetes cluster
  update      Update an existing Kubernetes cluster
  validate    Validate cluster assets
  version     Print version information and exit

Use "kube-aws [command] --help" for more information about a command.

Was hoping I might be able to use the older version to delete my node-pools created with a newer version of kube-aws (that seems to be missing the node-pool sub-command), but seems like perhaps the node-pools configuration has changed and now it is in the main cluster.yaml file versus in sub directories

/usr/local/bin/kube-aws-v0.9.3-rc.5 node-pools destroy --node-pool-name nodepool1
Error: Error parsing node pool config: open node-pools/nodepool1/cluster.yaml: no such file or directory

To move forward created the nested sub-dirs and copied the cluster.yaml file into the tree and then deleted the node-pools again with the older kube-aws version

mkdir -p node-pools/nodepool1/
cp cluster.yaml !$
/usr/local/bin/kube-aws-v0.9.3-rc.5 node-pools destroy --node-pool-name nodepool1
CloudFormation stack is being destroyed. This will take several minutes

[mumoshu]

Hi @cmcconnell1, thanks as always for trying kube-aws!
Sorry about the documentation - it seems that we've missed updating the doc before tagging the release. I guess you can consult the latest documentation in the master branch without problems as of today.

Anyway, since v0.9.5, kube-aws node-pools sub-command is removed and integrated into the top-level up, update and destroy commands via #315.
Basically, you can now define your node pool(s) under the worker.nodePools[] array in cluster.yaml and then running kube-aws up creates all the cfn stacks including a control-plane and node pool(s).

[cmcconnell1]

Hello @mumoshu
Thanks for the quick response.

AIR, I initially did try to just do a kube-aws destroy but wound up with errors within the cloudformation API, that's when I checked the doc's and went down that route.
Looking back from earlier today AM, I think these are the initial errors from the
kube-aws destroy before trying the node-pool option sub-command:

13:38:22 UTC-0700	DELETE_FAILED	AWS::CloudFormation::Stack	ops-dev	The following resource(s) failed to delete: [Nodepool1, Controlplane].
13:38:21 UTC-0700	DELETE_FAILED	AWS::CloudFormation::Stack	Controlplane	Embedded stack arn:aws:cloudformation:us-west-1:076613928512:stack/ops-dev-Controlplane-1LCJW7NEY3OY8/7e0128e0-1641-11e7-b531-500c2171aec6 was not successfully deleted: Export ops-dev-Controlplane-1LCJW7NEY3OY8-Etcd0EIP cannot be deleted as it is in use by ops-dev-Nodepool1-GNSGPW1EUPS
13:38:10 UTC-0700	DELETE_IN_PROGRESS	AWS::CloudFormation::Stack	Controlplane	
13:38:08 UTC-0700	DELETE_FAILED	AWS::CloudFormation::Stack	Nodepool1	Embedded stack arn:aws:cloudformation:us-west-1:076613928512:stack/ops-dev-Nodepool1-GNSGPW1EUPS/2b362c80-1643-11e7-a1c3-500cf8ee6035 was not successfully deleted: The following resource(s) failed to delete: [IAMRoleWorker].
13:37:57 UTC-0700	DELETE_IN_PROGRESS	AWS::CloudFormation::Stack	Nodepool1	
13:37:55 UTC-0700	DELETE_IN_PROGRESS	AWS::CloudFormation::Stack	ops-dev	User Initiated
12:07:44 UTC-0700	DELETE_FAILED	AWS::CloudFormation::Stack	ops-dev	The following resource(s) failed to delete: [Nodepool1, Controlplane].
12:07:43 UTC-0700	DELETE_FAILED	AWS::CloudFormation::Stack	Controlplane	Embedded stack arn:aws:cloudformation:us-west-1:076613928512:stack/ops-dev-Controlplane-1LCJW7NEY3OY8/7e0128e0-1641-11e7-b531-500c2171aec6 was not successfully deleted: Export ops-dev-Controlplane-1LCJW7NEY3OY8-OpsDevIgwPublic1b cannot be deleted as it is in use by ops-dev-Nodepool1-GNSGPW1EUPS
12:07:31 UTC-0700	DELETE_IN_PROGRESS	AWS::CloudFormation::Stack	Controlplane	
12:07:28 UTC-0700	DELETE_FAILED	AWS::CloudFormation::Stack	Nodepool1	Embedded stack arn:aws:cloudformation:us-west-1:076613928512:stack/ops-dev-Nodepool1-GNSGPW1EUPS/2b362c80-1643-11e7-a1c3-500cf8ee6035 was not successfully deleted: The following resource(s) failed to delete: [IAMRoleWorker].
12:05:52 UTC-0700	DELETE_IN_PROGRESS	AWS::CloudFormation::Stack	Nodepool1	
12:05:49 UTC-0700	DELETE_IN_PROGRESS	AWS::CloudFormation::Stack	ops-dev	User Initiated

I got in a bit of a mess by manually trying to purge, detach, disassociate resources that I saw in the CF event logs.
Will pick back up on this tomorrow AM.
Thanks

[cmcconnell1]

Another note for anyone getting stuck with orphaned kube cluster infrastructure components

(especially those using deis workflow) is that you will want to delete and un-munge anything done outside of kube-aws--including IAM roles you might have applied to cluster resources; your deis ELB, etc., before destroying your kube cluster.

Else the Deis ELB persists and has a network interface associated and it's contained within a kube cluster subnet. So these will fail to delete and cause your cnf stack to fail upon delete, and ultimately prevents any clean kube-aws destroy from completing.
A tool that I seem to not be able to live without now is awless, which is so much more useful to show relationships and help track down dependencies, cruft, etc.

awless show subnet-123ae456
| PROPERTY ▲ |           VALUE           |
|------------|---------------------------|
| CIDR       | 10.1.17.0/24              |
| Default    | false                     |
| ID         | subnet-123ae456           |
| Name       | ops-dev-OpsDevIgwPublic1b |
| Public     | true                      |
| State      | available                 |
| Vpc        | vpc-abc12ca1              |
| Zone       | us-west-1b                |

# Relations:
us-west-1[region]
	↳ @terradatum-main[vpc]
		↳ @ops-dev-OpsDevIgwPublic1b[subnet]

Siblings: @EXTERNAL-1A[subnet], @EXTERNAL-1B[subnet], @INTERNAL-1A[subnet], ... (display all with flag --siblings)

[mumoshu]

Note: Confirmed to work with v0.9.6-rc.2 according to #189 (comment)

Thanks for your confirmation @cmcconnell1 👍

[mumoshu]

Also - thanks for the tips! I believe it should be noted in our documentation. I'll open an another issue to track the documentation.