waiting for CloudFormation stack to reach "DELETE_COMPLETE" status: RequestCanceled: waiter context canceled caused by: context deadline exceeded #843

rafaelvanoni · 2019-06-05T22:32:43Z

I've been running into AWS timeouts when trying to delete a cluster (see output below) even with the wait option.

I took a look at the eksctl source and I'm wondering if we could set api.DefaultWaitTimeout to an "infinite" value when the -w option is passed. I'd be happy to propose a PR if that makes sense to the dev team. The problem this causes is that I have to go into the web interface and delete the stack(s) manually.

+ eksctl delete cluster -w --name=wt-dns-policy-k8s-aws-cni-etcd-232 --region us-west-2
[ℹ]  using region us-west-2
[ℹ]  deleting EKS cluster "wt-dns-policy-k8s-aws-cni-etcd-232"
[✔]  kubeconfig has been updated
[ℹ]  2 sequential tasks: { delete nodegroup "ng-d8c302eb", delete cluster control plane "wt-dns-policy-k8s-aws-cni-etcd-232" }
[ℹ]  will delete stack "eksctl-wt-dns-policy-k8s-aws-cni-etcd-232-nodegroup-ng-d8c302eb"
[ℹ]  waiting for stack "eksctl-wt-dns-policy-k8s-aws-cni-etcd-232-nodegroup-ng-d8c302eb" to get deleted
[ℹ]  will delete stack "myCluster"
[ℹ]  waiting for stack "myCluster" to get deleted
[✖]  unexpected status "DELETE_IN_PROGRESS" while waiting for CloudFormation stack "myCluster"
[ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2B: DELETE_COMPLETE
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2A: DELETE_COMPLETE
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2B: DELETE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: DELETE_COMPLETE
[ℹ]  AWS::IAM::Role/ServiceRole: DELETE_COMPLETE
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2A: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2B: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2A: DELETE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/ServiceRole: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2B: DELETE_IN_PROGRESS
[ℹ]  AWS::EKS::Cluster/ControlPlane: DELETE_COMPLETE
[ℹ]  AWS::EC2::InternetGateway/InternetGateway: DELETE_COMPLETE
[ℹ]  AWS::EC2::InternetGateway/InternetGateway: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::VPCGatewayAttachment/VPCGatewayAttachment: DELETE_COMPLETE
[ℹ]  AWS::EC2::EIP/NATIP: DELETE_COMPLETE
[ℹ]  AWS::EC2::EIP/NATIP: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::NatGateway/NATGateway: DELETE_COMPLETE
[ℹ]  AWS::EC2::RouteTable/PrivateRouteTable: DELETE_COMPLETE
[ℹ]  AWS::EC2::RouteTable/PublicRouteTable: DELETE_COMPLETE
[ℹ]  AWS::EC2::NatGateway/NATGateway: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::RouteTable/PrivateRouteTable: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::RouteTable/PublicRouteTable: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::Route/PrivateSubnetRoute: DELETE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2A: DELETE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2B: DELETE_COMPLETE
[ℹ]  AWS::EC2::Route/PublicSubnetRoute: DELETE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2B: DELETE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2A: DELETE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroup/ClusterSharedNodeSecurityGroup: DELETE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroup/ClusterSharedNodeSecurityGroup: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::SecurityGroupIngress/IngressInterNodeGroupSG: DELETE_COMPLETE
[ℹ]  AWS::IAM::Policy/PolicyCloudWatchMetrics: DELETE_COMPLETE
[ℹ]  AWS::IAM::Policy/PolicyNLB: DELETE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroupIngress/IngressInterNodeGroupSG: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2A: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2B: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::Route/PrivateSubnetRoute: DELETE_IN_PROGRESS
[ℹ]  AWS::IAM::Policy/PolicyNLB: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2B: DELETE_IN_PROGRESS
[ℹ]  AWS::EKS::Cluster/ControlPlane: DELETE_IN_PROGRESS
[ℹ]  AWS::IAM::Policy/PolicyCloudWatchMetrics: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::Route/PublicSubnetRoute: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2A: DELETE_IN_PROGRESS
[ℹ]  AWS::EC2::VPCGatewayAttachment/VPCGatewayAttachment: DELETE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/myCluster: DELETE_IN_PROGRESS – "User Initiated"
[ℹ]  AWS::CloudFormation::Stack/myCluster: CREATE_COMPLETE
[ℹ]  AWS::EKS::Cluster/ControlPlane: CREATE_COMPLETE
[ℹ]  AWS::EC2::Route/PrivateSubnetRoute: CREATE_COMPLETE
[ℹ]  AWS::EC2::Route/PrivateSubnetRoute: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::Route/PrivateSubnetRoute: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::NatGateway/NATGateway: CREATE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2A: CREATE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2B: CREATE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2B: CREATE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2A: CREATE_COMPLETE
[ℹ]  AWS::EKS::Cluster/ControlPlane: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::Route/PublicSubnetRoute: CREATE_COMPLETE
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2B: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2A: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2B: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EKS::Cluster/ControlPlane: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::NatGateway/NATGateway: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2A: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2B: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2B: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPublicUSWEST2A: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::NatGateway/NATGateway: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::SubnetRouteTableAssociation/RouteTableAssociationPrivateUSWEST2A: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2B: CREATE_COMPLETE
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2B: CREATE_COMPLETE
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2A: CREATE_COMPLETE
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2A: CREATE_COMPLETE
[ℹ]  AWS::EC2::VPCGatewayAttachment/VPCGatewayAttachment: CREATE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroupIngress/IngressInterNodeGroupSG: CREATE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroupIngress/IngressInterNodeGroupSG: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SecurityGroupIngress/IngressInterNodeGroupSG: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_COMPLETE
[ℹ]  AWS::EC2::SecurityGroup/ClusterSharedNodeSecurityGroup: CREATE_COMPLETE
[ℹ]  AWS::EC2::Route/PublicSubnetRoute: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SecurityGroup/ClusterSharedNodeSecurityGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::Route/PublicSubnetRoute: CREATE_IN_PROGRESS
[ℹ]  AWS::IAM::Policy/PolicyCloudWatchMetrics: CREATE_COMPLETE
[ℹ]  AWS::IAM::Policy/PolicyNLB: CREATE_COMPLETE
[ℹ]  AWS::EC2::RouteTable/PublicRouteTable: CREATE_COMPLETE
[ℹ]  AWS::EC2::RouteTable/PrivateRouteTable: CREATE_COMPLETE
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2B: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::RouteTable/PublicRouteTable: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2B: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::RouteTable/PrivateRouteTable: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2A: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::SecurityGroup/ControlPlaneSecurityGroup: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::VPCGatewayAttachment/VPCGatewayAttachment: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2A: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::RouteTable/PublicRouteTable: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2B: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::RouteTable/PrivateRouteTable: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2B: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::Subnet/SubnetPrivateUSWEST2A: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::SecurityGroup/ClusterSharedNodeSecurityGroup: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::VPCGatewayAttachment/VPCGatewayAttachment: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::Subnet/SubnetPublicUSWEST2A: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::VPC/VPC: CREATE_COMPLETE
[ℹ]  AWS::EC2::EIP/NATIP: CREATE_COMPLETE
[ℹ]  AWS::EC2::InternetGateway/InternetGateway: CREATE_COMPLETE
[ℹ]  AWS::IAM::Policy/PolicyCloudWatchMetrics: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Policy/PolicyNLB: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Policy/PolicyCloudWatchMetrics: CREATE_IN_PROGRESS
[ℹ]  AWS::IAM::Policy/PolicyNLB: CREATE_IN_PROGRESS
[ℹ]  AWS::IAM::Role/ServiceRole: CREATE_COMPLETE
[ℹ]  AWS::EC2::VPC/VPC: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::EIP/NATIP: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Role/ServiceRole: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::EC2::VPC/VPC: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::EIP/NATIP: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::InternetGateway/InternetGateway: CREATE_IN_PROGRESS – "Resource creation Initiated"
[ℹ]  AWS::IAM::Role/ServiceRole: CREATE_IN_PROGRESS
[ℹ]  AWS::EC2::InternetGateway/InternetGateway: CREATE_IN_PROGRESS
[ℹ]  AWS::CloudFormation::Stack/myCluster: CREATE_IN_PROGRESS – "User Initiated"
[ℹ]  1 error(s) occurred while deleting cluster with nodegroup(s)
[✖]  waiting for CloudFormation stack "myCluster" to reach "DELETE_COMPLETE" status: RequestCanceled: waiter context canceled
caused by: context deadline exceeded
[✖]  failed to delete cluster with nodegroup(s)

The text was updated successfully, but these errors were encountered:

errordeveloper · 2019-06-06T12:30:09Z

You could try --timeout=999h or something like that. Also, you should try deleting it via CloudFormation console, and it might revile the root cause for this - it'd be good to determine what it is. We have various deletion issues (see area/deletions label), this one looks like a unique case.

rafaelvanoni · 2019-06-06T17:54:32Z

Thanks for the feedback, I'll retry through the console and report back.

rafaelvanoni · 2019-06-06T18:05:42Z

Just looked at a previous failure and this is the error that caused the delete to fail:

10:59:40 UTC-0700 | DELETE_FAILED | AWS::EC2::Subnet | SubnetPrivateUSWEST2B | The  subnet 'subnet-0b2782ebcee7a1122' has dependencies and cannot be  deleted. (Service: AmazonEC2; Status Code: 400; Error Code:  DependencyViolation; Request ID: ...

I'm running aws ec2 delete-vpc after the eksctl delete cluster as the vpc dependencies created by the cloud formation prevent me from doing it first.

errordeveloper · 2019-06-07T07:31:51Z

Have a look if there is an ELB in that VPC? We have #103 to address that, and I certainly would like to prioritise it.

rafaelvanoni · 2019-06-07T09:43:21Z

There isn't. I had an elasticache cache cluster and a cache cluster subnet on the VPC, but both are deleted before eksctl delete is run. I also tried adding --timeout and the operation still fails, so the problem might be somewhere else (and not just a matter of waiting longer).

errordeveloper · 2019-06-07T11:27:49Z

Ok, so if you try deleting the VPC via the console, it should give you a few hints.

…

On Fri, 7 Jun 2019, 10:43 am Rafael Vanoni, ***@***.***> wrote: There isn't. I had an elasticache cache cluster and a cache cluster subnet on the VPC, but both are deleted before eksctl delete is run. I also tried adding --timeout and the operation still fails, so the problem might be somewhere else (and not just a matter of waiting longer). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#843>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB5MS3K2M5NHJ5CUFFTS2TPZIUTVANCNFSM4HULYKFQ> .

rafaelvanoni · 2019-06-07T21:55:34Z

Just re-tried eksctl delete, it failed complaining that the VPC couldn't be deleted, but the VPC isn't listed in the AWS dashboard anymore.

rafaelvanoni · 2019-06-12T21:11:20Z

I'm still unable to delete the eks cluster, even though all of its resources are gone. Any ideas?

rafaelvanoni · 2019-06-12T23:31:15Z

Renaming this to more accurately indicate the error.

errordeveloper · 2019-06-13T07:39:03Z

Just re-tried eksctl delete, it failed complaining that the VPC couldn't be deleted, but the VPC isn't listed in the AWS dashboard anymore.

Ok, so you must have managed to delete the VPC. Please go to CloudFormation console and delete from there, if fails on the first attempt - try to leave the VPC out (it will offer a menu).

rafaelvanoni · 2019-07-22T20:38:04Z

I've moved on from work involving this issue, so if you can't reproduce it on your end please feel free to close it. Thanks for taking a look at it.

errordeveloper · 2019-07-23T06:33:26Z

Thanks for letting us know. This can be closed as it was addressed in #1010.

Fix broken gomocks

rafaelvanoni added the kind/feature New feature or request label Jun 5, 2019

errordeveloper added the area/deletions label Jun 6, 2019

rafaelvanoni changed the title ~~set timeout to infinity when delete cluster is used with the -w option~~ waiting for CloudFormation stack to reach "DELETE_COMPLETE" status: RequestCanceled: waiter context canceled caused by: context deadline exceeded Jun 12, 2019

errordeveloper closed this as completed Jul 23, 2019

torredil pushed a commit to torredil/eksctl that referenced this issue May 20, 2022

Merge pull request eksctl-io#843 from wongma7/fixmocks

44246b7

Fix broken gomocks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

waiting for CloudFormation stack to reach "DELETE_COMPLETE" status: RequestCanceled: waiter context canceled caused by: context deadline exceeded #843

waiting for CloudFormation stack to reach "DELETE_COMPLETE" status: RequestCanceled: waiter context canceled caused by: context deadline exceeded #843

rafaelvanoni commented Jun 5, 2019 •

edited by errordeveloper

Loading

errordeveloper commented Jun 6, 2019 •

edited

Loading

rafaelvanoni commented Jun 6, 2019

rafaelvanoni commented Jun 6, 2019 •

edited

Loading

errordeveloper commented Jun 7, 2019

rafaelvanoni commented Jun 7, 2019

errordeveloper commented Jun 7, 2019 via email

rafaelvanoni commented Jun 7, 2019 •

edited

Loading

rafaelvanoni commented Jun 12, 2019

rafaelvanoni commented Jun 12, 2019

errordeveloper commented Jun 13, 2019

rafaelvanoni commented Jul 22, 2019

errordeveloper commented Jul 23, 2019

waiting for CloudFormation stack to reach "DELETE_COMPLETE" status: RequestCanceled: waiter context canceled caused by: context deadline exceeded #843

waiting for CloudFormation stack to reach "DELETE_COMPLETE" status: RequestCanceled: waiter context canceled caused by: context deadline exceeded #843

Comments

rafaelvanoni commented Jun 5, 2019 • edited by errordeveloper Loading

errordeveloper commented Jun 6, 2019 • edited Loading

rafaelvanoni commented Jun 6, 2019

rafaelvanoni commented Jun 6, 2019 • edited Loading

errordeveloper commented Jun 7, 2019

rafaelvanoni commented Jun 7, 2019

errordeveloper commented Jun 7, 2019 via email

rafaelvanoni commented Jun 7, 2019 • edited Loading

rafaelvanoni commented Jun 12, 2019

rafaelvanoni commented Jun 12, 2019

errordeveloper commented Jun 13, 2019

rafaelvanoni commented Jul 22, 2019

errordeveloper commented Jul 23, 2019

rafaelvanoni commented Jun 5, 2019 •

edited by errordeveloper

Loading

errordeveloper commented Jun 6, 2019 •

edited

Loading

rafaelvanoni commented Jun 6, 2019 •

edited

Loading

rafaelvanoni commented Jun 7, 2019 •

edited

Loading