Tool or job to cleanup ingress related GCP resource after test failure

[MrHohn]

Ref #438 (comment). Ingress scale test occasionally will hit quota issue due to resource leaked by previous runs / test failures. Test failures can be caused by various reasons --- new change breaks controller (e.g. nil pointer), test timeout, test infrastructure problem (e.g. run#858 never finish), etc.

Manually deleting leftover resource every time is not tenable. We should at least have a cleanup script we can run. More preferable if an automated job can run it.

@krzyzacy Is this something we should put on https://k8s-testgrid.appspot.com/sig-testing-misc#ci-janitor? Or we should have our own janitor because this is ingress specific?

[krzyzacy]

if you can fit into the current janitor script, sure :-)

[MrHohn]

Humm, https://github.com/kubernetes/test-infra/blob/master/boskos/janitor/janitor.py does include most of the ingress related resource, except SSL cert.

Though I'm not seeing it runs against ingress test projects; https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/maintenance-ci-janitor/584

We should probably enable it?

[MrHohn]

In fact, seems like ingress-project is included:
https://github.com/kubernetes/test-infra/blob/795cceb4ce411fa43933fb6534c208a6d470e1ab/boskos/janitor/deployment.yaml#L52

Though we don't put the ingress scale project under that category. I can create a new category "ingress-scale-project" and add it to the same place.

[krzyzacy]

oh, that's handled by boskos, not ci-janitor

(also ci-janitor is borked 🙃 fixing with kubernetes/test-infra#9486)

[krzyzacy]

it's handled by https://github.com/kubernetes/test-infra/blob/master/boskos/janitor/janitor.go

if you have access to k8s-prow you can check logs there, which project you are looking for?

[MrHohn]

@krzyzacy Thanks for the quick response, I were looking for k8s-ingress-e2e-scale-backup, which is not defined in any bosko resource yet.

BTW did you mean https://prow.k8s.io? Not actually sure which job to look at :)

[krzyzacy]

I mean https://pantheon.corp.google.com/home/dashboard?project=k8s-prow

if it's not in the pool then it will be handled by ci-janitor - let's wait for next run.

[MrHohn]

if it's not in the pool then it will be handled by ci-janitor - let's wait for next run.

Thanks! Will check on the next run. And unfortunately I don't have access to k8s-prow :/

[krzyzacy]

time="2018-09-20T04:48:12Z" level=info msg="Will clean up k8s-ingress-e2e-scale-backup with ttl 24 h" 
time="2018-09-20T04:48:38Z" level=info msg="successfully cleaned up project k8s-ingress-e2e-scale-backup" 

it should be cleaned up, double check the project? if something is still lingering then we should add them to the janitor script.

[MrHohn]

@krzyzacy Awesome, yeah seems like it should work. Good to know this is already covered :)

Closing this issue. I will send follow up to cleanup SSL cert as well.