Make critical jobs Guaranteed Pod QOS: ci-kubernetes-build

[spiffxp]

What should be cleaned up or changed:

This is part of #18530

The following jobs should be Guaranteed Pod QOS, meaning they should have CPU and memory resource limits, and matching resource requests:

• ci-kubernetes-build
• ci-kubernetes-build-1-19
• ci-kubernetes-build-stable1
• ci-kubernetes-build-stable2
• ci-kubernetes-build-stable3

These jobs run on (google.com only) k8s-prow-build, so @spiffxp has provided the following guess:

• 7 cpu (or as near 8 as you can get), slightly above 32 Gi mem (34?)

General steps to follow:

• update the job definitions in /config/jobs to have matching CPU and memory limits and requests
  • e.g. (ci-kubernetes-e2e-gci-gce has a resources: field with matching entries)
• open a pull request, include a link to this issue and cc @kubernetes/ci-signal in the description
• keep an eye on the jobs for the next few days, and if they start failing more than usual, open followup pull requests to raise resources
  • can look at recent prow.k8s.io runs (e.g. https://prow.k8s.io/?job=ci-kubernetes-e2e-gci-gce)
  • can look at job history (e.g. https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce)
  • can look at testgrid dashboards (e.g. https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-default)
• after the job has remained stable for a few days, declare victory

/sig testing
/sig release
/area jobs
/area release-eng

[helenfeng737]

/assign

[helenfeng737]

https://prow.k8s.io/?job=ci-kubernetes-build
https://prow.k8s.io/?job=ci-kubernetes-build-1-19
https://prow.k8s.io/?job=ci-kubernetes-build-stable1
https://prow.k8s.io/?job=ci-kubernetes-build-stable2
https://prow.k8s.io/?job=ci-kubernetes-build-stable3

https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-build
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-build-1-19
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-build-stable1
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-build-stable2
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-build-stable3

[spiffxp]

/remove-help
since @ZhiFeng1993 has it

[helenfeng737]

/close

I think it's safe to close this one. cc. @RobertKielty

[k8s-ci-robot]

@ZhiFeng1993: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

/close

I think it's safe to close this one. cc. @RobertKielty

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

Looking at testgrids

https://testgrid.k8s.io/sig-release-master-blocking#build-master&graph-metrics=test-duration-minutes&width=20 - looks mostly ok

https://testgrid.k8s.io/sig-release-1.19-blocking#build-1.19&graph-metrics=test-duration-minutes&width=20 - same

https://testgrid.k8s.io/sig-release-1.18-blocking#build-1.18&graph-metrics=test-duration-minutes&width=20 - this concerns me

The two peaks on the right are what I think is "good" behavior... the job fails due to timeout for Reasons™, but there is a followup run that doesn't, and the build is available for use.

The two peaks on the left don't have any runs after that are OK. Did incomplete builds get published?

More importantly... is this new behavior? Or has this been happening prior to us setting resource constraints?

[spiffxp]

The resource limit changes merged 2020-08-03 4:20pm PDT

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-build-stable1/1289866689743163392 - is an example of a build falling prey to this problem before the change was deployed. So I don't think this is new behavior

[spiffxp]

I opened #18808 to address the bad build job behavior

Given that it's pre-existing bad behavior, I'm willing to call this done

/close

[spiffxp]

Thanks @ZhiFeng1993 !

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

I opened #18808 to address the bad build job behavior

Given that it's pre-existing bad behavior, I'm willing to call this done

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.