[GLBC] GCE resources of non-snapshotted ingresses are not deleted

[bowei]

From @nicksardo on March 13, 2017 20:34

For the GLBC to know about load balancer GCE resources, it must first have snapshotted an ingress object that created those resources. This happens when an ingress object is deleted while the GLBC is offline or starting up.

The following are test logs from gce-gci-latest-upgrade-etcd:

Successful Test #334 : glbc.log

I0313 19:04:50.100770       5 utils.go:166] Syncing e2e-tests-ingress-upgrade-pjrvv/static-ip
... # GLBC has listed existing ingress objects and knows about the `static-ip` ingress
E0313 19:04:55.101688       5 utils.go:168] Requeuing e2e-tests-ingress-upgrade-pjrvv/static-ip, err Waiting for stores to sync
... # Ingress is being requeued because other stores have not finished syncing
I0313 19:04:55.107613       5 utils.go:166] Syncing e2e-tests-ingress-upgrade-pjrvv/static-ip
I0313 19:04:55.107669       5 controller.go:295] Syncing e2e-tests-ingress-upgrade-pjrvv/static-ip
... # Ingresses are listed from apiserver
I0313 19:04:55.594861       5 loadbalancers.go:165] Creating loadbalancers [0xc42067a2d0]
# Following the above line, the ingress obj will be added to the L7.snapshotter (cache)
I0313 19:04:55.755621       5 controller.go:128] Delete notification received for Ingress e2e-tests-ingress-upgrade-pjrvv/static-ip
# Ingress watcher notices the ingress object was deleted

Failed Test #332 : glbc.log

I0313 17:11:53.401594       5 utils.go:166] Syncing e2e-tests-ingress-upgrade-9shgs/static-ip
... 
I0313 17:11:54.570150       5 controller.go:128] Delete notification received for Ingress e2e-tests-ingress-upgrade-9shgs/static-ip
... # Watcher notices the ingress object has been deleted; however, we have not yet snapshotted the ingress object
E0313 17:11:58.401711       5 utils.go:168] Requeuing e2e-tests-ingress-upgrade-9shgs/static-ip, err Waiting for stores to sync
...
I0313 17:11:58.401773       5 controller.go:295] Syncing e2e-tests-ingress-upgrade-9shgs/static-ip
... # Ingresses are listed from apiserver but because the ingress was deleted, the list is empty
I0313 17:11:58.401852       5 loadbalancers.go:165] Creating loadbalancers []
# Ingress object is not created 

The L7Pool GC func deletes resources of ingresses stored in the l7.snapshotter that are not mentioned by name in the arg slice. Because the test ingress was never stored in the snapshot cache, the GCE resources are never deleted.

The failed test log also contains multiple blocks of the following:

I0313 17:12:27.161699       5 backends.go:336] GCing backend for port 32397
I0313 17:12:27.161767       5 backends.go:233] Deleting backend k8s-be-32397--179702b6ab620fd3
...
E0313 17:12:27.390153       5 utils.go:168] Requeuing e2e-tests-ingress-upgrade-9shgs/static-ip, err Error during sync <nil>, error during GC googleapi: Error 400: The backend_service resource 'k8s-be-32397--179702b6ab620fd3' is already being used by 'k8s-um-e2e-tests-ingress-upgrade-9shgs-static-ip--179702b6ab620', resourceInUseByAnotherResource

The GLBC knows about the extraneous backends because the BackendPool uses the CloudListingPool. This implementation calls a List func to reflect the current state of GCE.

Copied from original issue: kubernetes/ingress-nginx#431

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[nicksardo]

/remove-lifecycle rotten
/lifecycle frozen

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[rramkumar1]

/remove-lifecycle stale

[bowei]

/lifecycle frozen

[rramkumar1]

This was fixed by #590 and is shipped in v1.5.0.