Cleanup finalizer workflow and improve Ingress GC #825

skmatti · 2019-08-13T23:37:31Z

This fixes a bug on Ingress resource deletion when Ingress class changes from a GLBC resource to a non-GLBC one. Finalizer of a non-GLBC Ingress resource is removed when GC is invoked on it.

k8s-ci-robot · 2019-08-13T23:37:32Z

Welcome @skmatti!

It looks like this is your first PR to kubernetes/ingress-gce 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/ingress-gce has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2019-08-13T23:37:33Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2019-08-13T23:37:39Z

Hi @skmatti. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

MrHohn · 2019-08-13T23:52:08Z

/ok-to-test

skmatti · 2019-08-13T23:53:12Z

/assign @MrHohn

MrHohn

Thanks, just a few nits.

pkg/controller/controller.go

pkg/controller/controller_test.go

bowei · 2019-08-14T07:23:09Z

/hold

If the Ingress has a finalizer, that means we have potentially allocated resources associated with the previous state of the Ingress. Which means we need to keep the finalizer...

rramkumar1 · 2019-08-14T13:01:21Z

@bowei If you look at line 496 of controller.go, you can see that if the class was changed from glbc to non-glbc, then we add the Ingress for garbage collection to ensure we clean stuff up. This was to fix some bug not related to finalizer support.

I think we should add all non-glbc Ingresses for garbage collection always (rather than just when the class is changed) as a sanity check that stuff is getting cleaned up.

bowei · 2019-08-14T19:52:00Z

/hold cancel

bowei · 2019-08-14T20:03:34Z

Discussed offline -- looks like the finalizer handling is currently incorrect and needs to be fixed.

bowei · 2019-08-14T20:03:50Z

/hold

bowei · 2019-08-15T00:27:40Z

/assign

pkg/backends/backends.go

pkg/controller/controller.go

pkg/healthchecks/healthchecks.go

pkg/utils/utils.go

pkg/utils/utils_test.go

bowei

discussed making the filtering and set of ingresses that the code is operating on more obvious

pkg/backends/backends.go

pkg/controller/controller.go

pkg/controller/controller_test.go

pkg/utils/utils_test.go

pkg/sync/interfaces.go

pkg/controller/controller.go

bowei · 2019-08-28T00:53:52Z

/lgtm
/approve

k8s-ci-robot · 2019-08-28T00:54:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bowei, MrHohn, skmatti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [MrHohn,bowei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bowei · 2019-08-28T00:55:33Z

/hold cancel

MrHohn · 2019-08-28T00:32:45Z

pkg/controller/controller.go

+				klog.Errorf("Invalid object type: %T", obj)
+				return
+			}
+			if delIng.ObjectMeta.DeletionTimestamp != nil {


Can we have some comments here to explain the reason behind? BTW have we verified that this for sure isn't the case when an Ingress is deleted without finalizer? (I'm still reading through https://github.com/kubernetes/apiserver/blob/master/pkg/registry/generic/registry/store.go...)

Yeah, I tested this behavior. Deletion timestamp is being added only with finalizer enabled. I think I added a comment, but later changed to a log comment below this line.

MrHohn · 2019-08-28T00:51:33Z

pkg/controller/controller.go

-			}
-		}
+// MaybeRemoveFinalizers cleans up Finalizers if needed.
+func (lbc *LoadBalancerController) MaybeRemoveFinalizers(state interface{}) error {


Why Maybe?

MrHohn · 2019-08-28T00:55:26Z

pkg/controller/types.go

-	lbNames   []string
-	svcPorts  []utils.ServicePort
+	// ingressesToCleanup is the list of ingressesToCleanup that needs to be cleaned up during GC.
+	ingressesToCleanup []*v1beta1.Ingress


I vaguely remember we uses this as a signal for whether we can remove the instance group (if there is no ingresses exist). Has that logic been changed?

Ahh, seems like we look at lbNames instead.

ingress-gce/pkg/controller/controller.go

Lines 385 to 392 in fddd91d

// TODO(ingress#120): Move this to the backend pool so it mirrors creation

if len(gcState.lbNames) == 0 {

igName := lbc.ctx.ClusterNamer.InstanceGroup()

klog.Infof("Deleting instance group %v", igName)

if err := lbc.instancePool.DeleteInstanceGroup(igName); err != err {

return err

}

}

So we need to keep multi-cluster ingress in this case then? :/

We are using lbnames to see if we can delete instance groups.

skmatti · 2019-10-01T21:10:29Z

pkg/controller/controller.go

+func toLbNames(ings []*v1beta1.Ingress) []string {
+	lbNames := make([]string, 0, len(ings))
+	for _, ing := range ings {
+		lbNames = append(lbNames, utils.IngressKeyFunc(ing))


IngressKeyFunc used here is incorrect. It should be replaced with KeyFunc defined here.

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Aug 13, 2019

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 13, 2019

k8s-ci-robot requested review from bowei and rramkumar1 August 13, 2019 23:37

k8s-ci-robot assigned MrHohn Aug 13, 2019

MrHohn approved these changes Aug 14, 2019

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 14, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 14, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 14, 2019

skmatti changed the title ~~Remove finalizer on changing Ingress class to non-GLBC~~ wip: Remove finalizer on changing Ingress class to non-GLBC Aug 15, 2019

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 15, 2019

skmatti changed the title ~~wip: Remove finalizer on changing Ingress class to non-GLBC~~ WIP: Remove finalizer on changing Ingress class to non-GLBC Aug 15, 2019

k8s-ci-robot assigned bowei Aug 15, 2019

k8s-ci-robot removed the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 16, 2019

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 21, 2019

skmatti force-pushed the release-1.6 branch 3 times, most recently from c88c5b7 to 45a423c Compare August 22, 2019 07:19

bowei changed the title ~~Remove finalizer on changing Ingress class to non-GLBC~~ Cleanup finalizer workflow and improve Ingress GC Aug 22, 2019

bowei reviewed Aug 23, 2019

View reviewed changes

skmatti force-pushed the release-1.6 branch 2 times, most recently from 939f0c6 to 3f786ec Compare August 23, 2019 19:48

bowei reviewed Aug 24, 2019

View reviewed changes

skmatti force-pushed the release-1.6 branch 4 times, most recently from fce6b2f to 348537c Compare August 26, 2019 21:45

bowei reviewed Aug 27, 2019

View reviewed changes

skmatti force-pushed the release-1.6 branch from 348537c to 0fba64e Compare August 27, 2019 05:23

bowei reviewed Aug 27, 2019

View reviewed changes

pkg/sync/interfaces.go Outdated Show resolved Hide resolved

bowei reviewed Aug 27, 2019

View reviewed changes

pkg/controller/controller.go Outdated Show resolved Hide resolved

Cleanup finalizer workflow and improve Ingress GC

78801b3

skmatti force-pushed the release-1.6 branch from 0fba64e to 78801b3 Compare August 28, 2019 00:13

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 28, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 28, 2019

MrHohn reviewed Aug 28, 2019

View reviewed changes

k8s-ci-robot merged commit 52428d2 into kubernetes:master Aug 28, 2019

MrHohn mentioned this pull request Aug 28, 2019

Fix or verify the GC logic for multi-cluster ingress #836

Closed

skmatti commented Oct 1, 2019

View reviewed changes

skmatti mentioned this pull request Oct 1, 2019

Bug: Incorrect ingress key function is being used #879

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanup finalizer workflow and improve Ingress GC #825

Cleanup finalizer workflow and improve Ingress GC #825

skmatti commented Aug 13, 2019

k8s-ci-robot commented Aug 13, 2019

k8s-ci-robot commented Aug 13, 2019

k8s-ci-robot commented Aug 13, 2019

MrHohn commented Aug 13, 2019

skmatti commented Aug 13, 2019

MrHohn left a comment

bowei commented Aug 14, 2019

rramkumar1 commented Aug 14, 2019 •

edited

Loading

bowei commented Aug 14, 2019

bowei commented Aug 14, 2019

bowei commented Aug 14, 2019

bowei commented Aug 15, 2019

bowei left a comment

bowei commented Aug 28, 2019

k8s-ci-robot commented Aug 28, 2019

bowei commented Aug 28, 2019

MrHohn Aug 28, 2019

skmatti Aug 28, 2019

MrHohn Aug 28, 2019

MrHohn Aug 28, 2019

MrHohn Aug 28, 2019

skmatti Aug 28, 2019

skmatti Oct 1, 2019

	// TODO(ingress#120): Move this to the backend pool so it mirrors creation
	if len(gcState.lbNames) == 0 {
	igName := lbc.ctx.ClusterNamer.InstanceGroup()
	klog.Infof("Deleting instance group %v", igName)
	if err := lbc.instancePool.DeleteInstanceGroup(igName); err != err {
	return err
	}
	}

Cleanup finalizer workflow and improve Ingress GC #825

Cleanup finalizer workflow and improve Ingress GC #825

Conversation

skmatti commented Aug 13, 2019

k8s-ci-robot commented Aug 13, 2019

k8s-ci-robot commented Aug 13, 2019

k8s-ci-robot commented Aug 13, 2019

MrHohn commented Aug 13, 2019

skmatti commented Aug 13, 2019

MrHohn left a comment

Choose a reason for hiding this comment

bowei commented Aug 14, 2019

rramkumar1 commented Aug 14, 2019 • edited Loading

bowei commented Aug 14, 2019

bowei commented Aug 14, 2019

bowei commented Aug 14, 2019

bowei commented Aug 15, 2019

bowei left a comment

Choose a reason for hiding this comment

bowei commented Aug 28, 2019

k8s-ci-robot commented Aug 28, 2019

bowei commented Aug 28, 2019

MrHohn Aug 28, 2019

Choose a reason for hiding this comment

skmatti Aug 28, 2019

Choose a reason for hiding this comment

MrHohn Aug 28, 2019

Choose a reason for hiding this comment

MrHohn Aug 28, 2019

Choose a reason for hiding this comment

MrHohn Aug 28, 2019

Choose a reason for hiding this comment

skmatti Aug 28, 2019

Choose a reason for hiding this comment

skmatti Oct 1, 2019

Choose a reason for hiding this comment

rramkumar1 commented Aug 14, 2019 •

edited

Loading